首页 / HADOOP / 2.1 Hadoop Eclipse Plugin 配置及安装

2.1 Hadoop Eclipse Plugin 配置及安装

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了2.1 Hadoop Eclipse Plugin 配置及安装，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含4787字，纯文字阅读大概需要7分钟。

内容图文

Hadoop Eclipse 开发工具 主要分为

1、根据Hadoop版本生成插件
 
2、安装Hadoop Eclipse插件
 
3、配置Hadoop目录
 
4、配置Hadoop连接
 
5、新一个MapReduce工程
 
WordCount.java
MapReduce——WordCount问题总结
正成功输入出后信息:

14/05/21 23:06:47 INFO input.FileInputFormat: Total input paths to process : 2
14/05/21 23:06:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/05/21 23:06:47 WARN snappy.LoadSnappy: Snappy native library not loaded
14/05/21 23:06:48 INFO mapred.JobClient: Running job: job_201405220635_0009
14/05/21 23:06:49 INFO mapred.JobClient: map 0% reduce 0%
14/05/21 23:06:59 INFO mapred.JobClient: map 50% reduce 0%
14/05/21 23:07:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/21 23:07:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/21 23:07:11 INFO mapred.JobClient: map 100% reduce 100%
14/05/21 23:07:13 INFO mapred.JobClient: Job complete: job_201405220635_0009
14/05/21 23:07:13 INFO mapred.JobClient: Counters: 29
14/05/21 23:07:13 INFO mapred.JobClient: Job Counters
14/05/21 23:07:13 INFO mapred.JobClient: Launched reduce tasks=1
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=17386
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Launched map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: Data-local map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=12160
14/05/21 23:07:13 INFO mapred.JobClient: File Output Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Written=15
14/05/21 23:07:13 INFO mapred.JobClient: FileSystemCounters
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_READ=52
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_READ=252
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=177419
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=15
14/05/21 23:07:13 INFO mapred.JobClient: File Input Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Read=22
14/05/21 23:07:13 INFO mapred.JobClient: Map-Reduce Framework
14/05/21 23:07:13 INFO mapred.JobClient: Map output materialized bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Map input records=2
14/05/21 23:07:13 INFO mapred.JobClient: Reduce shuffle bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Spilled Records=8
14/05/21 23:07:13 INFO mapred.JobClient: Map output bytes=38
14/05/21 23:07:13 INFO mapred.JobClient: CPU time spent (ms)=5610
14/05/21 23:07:13 INFO mapred.JobClient: Total committed heap usage (bytes)=336338944
14/05/21 23:07:13 INFO mapred.JobClient: Combine input records=4
14/05/21 23:07:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=230
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input records=4
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input groups=2
14/05/21 23:07:13 INFO mapred.JobClient: Combine output records=4
14/05/21 23:07:13 INFO mapred.JobClient: Physical memory (bytes) snapshot=428146688
14/05/21 23:07:13 INFO mapred.JobClient: Reduce output records=2
14/05/21 23:07:13 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2140233728
14/05/21 23:07:13 INFO mapred.JobClient: Map output records=4

 

 

错误解决方法：

 

14/05/21 23:14:11 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/.staging/job_201405220635_0010
14/05/21 23:14:11 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:93)

 

cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists

 

说明目录已经存在了，需要删除后再执行，命令如下：

hadoop fs -rmr newout

 

原文：http://www.cnblogs.com/scwanglijun/p/3742617.html

内容总结

以上是互联网集市为您收集整理的2.1 Hadoop Eclipse Plugin 配置及安装全部内容，希望文章能够帮你解决2.1 Hadoop Eclipse Plugin 配置及安装所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

扫描二维码推送至手机访问。

本文链接：https://qyyshop.com/info/1096560.html

来源：【匿名】