Nutch的配置(使用MySQL作为数据存储)
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了Nutch的配置(使用MySQL作为数据存储),小编现在分享给大家,供广大互联网技能从业者学习和参考。文章包含4935字,纯文字阅读大概需要8分钟。
内容图文
![Nutch的配置(使用MySQL作为数据存储)](/upload/InfoBanner/zyjiaocheng/495/8e45e1ecd9ba4997ac6e3e61095c3cda.jpg)
这里假定nutch的根目录为:${APACHE_NUTCH_HOME}
配置${APACHE_NUTCH_HOME}/ivy/ivy.xml,确保Nutch使用MySQL作为数据存储
将[html] view plain copy print?- <dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>
<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>改成[html] view plain copy print?
- <dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>
<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>取消以下行的注释[html] view plain copy print?
- <dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />
<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />取消以下的行注释使用Mysql作为gora存储[html] view plain copy print?
- <!-- Uncomment this to use MySQL as database with SQL as Gora store. -->
- <dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>
<!-- Uncomment this to use MySQL as database with SQL as Gora store. --><dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>
编辑${APACHE_NUTCH_HOME}/conf/gora.properties
添加以下代码激活MySQL的配置
[html] view plain copy print?- ###############################
- # MySQL properties #
- ###############################
- gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
- gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
- gora.sqlstore.jdbc.user=xxxxx
- gora.sqlstore.jdbc.password=xxxxx
################################ MySQL properties ################################gora.sqlstore.jdbc.driver=com.mysql.jdbc.Drivergora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=truegora.sqlstore.jdbc.user=xxxxxgora.sqlstore.jdbc.password=xxxxx编辑 ${APACHE_NUTCH_HOME}/conf/gora-sql-mapping.xml ,将主键的长度由512改成767
[html] view plain copy print?
- <primarykey column="id" length="767"/>
<primarykey column="id" length="767"/>
配置${APACHE_NUTCH_HOME}/conf/nutch-site.xml
在 http.agent.name字段下增加一个名字,可以是任意值但不能为空! 如果需要的话可以添加额外的语言(例如en为英语),同时也可以设置默认编码格式为utf-8
[html] view plain copy print?- <property>
- <name>http.agent.name</name>
- <value>YourNutchSpider</value>
- </property>
- <property>
- <name>http.accept.language</name>
- <value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
- <description>Value of the "Accept-Language" request header field.
- This allows selecting non-English language as default one to retrieve.
- It is a useful setting for search engines build for certain national group.
- </description>
- </property>
- <property>
- <name>parser.character.encoding.default</name>
- <value>utf-8</value>
- <description>The character encoding to fall back to when no other information
- is available</description>
- </property>
- <property>
- <name>storage.data.store.class</name>
- <value>org.apache.gora.sql.store.SqlStore</value>
- <description>The Gora DataStore class for storing and retrieving data.
- Currently the following stores are available: ....
- </description>
- </property>
<property><name>http.agent.name</name><value>YourNutchSpider</value></property><property><name>http.accept.language</name><value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value><description>Value of the "Accept-Language" request header field.This allows selecting non-English language as default one to retrieve.It is a useful setting for search engines build for certain national group.</description></property><property><name>parser.character.encoding.default</name><value>utf-8</value><description>The character encoding to fall back to when no other informationis available</description></property><property><name>storage.data.store.class</name><value>org.apache.gora.sql.store.SqlStore</value><description>The Gora DataStore class for storing and retrieving data.Currently the following stores are available: ....</description></property>
在命令行下输入
sudo apt-get install ant
安装配置ant在命令行界面使用cd切换到nutch的根目录
可以在终端中输入以下指令开始你的第一个爬虫工作
[html] view plain copy print?- cd ${APACHE_NUTCH_HOME}/runtime/local
- mkdir -p urls
- echo ‘http://nutch.apache.org/‘ > urls/seed.txt
cd ${APACHE_NUTCH_HOME}/runtime/localmkdir -p urlsecho ‘http://nutch.apache.org/‘ > urls/seed.txt
Nutch 2.2使用以下命令开始爬虫,设置线程数为30[html] view plain copy print?
- bin/nutch crawl urls -threads 30
bin/nutch crawl urls -threads 30
要查看爬取的数据时,进入数据库中输入以下指令即可查看
mysql -u xxxxx -p
use nutch;
SELECT * FROM nutch.webpage;
翻译源:http://wiki.apache.org/nutch/#Nutch_2.X_tutorial.28s.29
Nutch的配置(使用MySQL作为数据存储)
标签:and ted name alt inline track 编码格式 根目录 mod
本文系统来源:http://www.cnblogs.com/jpfss/p/7904340.html
内容总结
以上是互联网集市为您收集整理的Nutch的配置(使用MySQL作为数据存储)全部内容,希望文章能够帮你解决Nutch的配置(使用MySQL作为数据存储)所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错,欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 gblab@vip.qq.com 举报,一经查实,本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。