hadoopwordcount新API例子
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了hadoopwordcount新API例子,小编现在分享给大家,供广大互联网技能从业者学习和参考。文章包含3829字,纯文字阅读大概需要6分钟。
内容图文
准备 准备一些输入文件,可以用hdfs dfs -put xxx/*?/user/fatkun/input上传文件 代码 package com.fatkun;?import java.io.IOException;import java.util.ArrayList;import java.util.List;import java.util.StringTokenizer;?import org.apache.commons.lo
准备
准备一些输入文件,可以用hdfs dfs -put xxx/*?/user/fatkun/input上传文件
代码
package com.fatkun; ? import java.io.IOException; import java.util.ArrayList; import java.util.List; import java.util.StringTokenizer; ? import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; ? public class WordCount extends Configured implements Tool { static enum Counters { INPUT_WORDS // 计数器 } ? static Log logger = LogFactory.getLog(WordCount.class); ? public static class CountMapper extends Mapper { private final IntWritable one = new IntWritable(1); private Text word = new Text(); private boolean caseSensitive = true; ? @Override protected void setup(Context context) throws IOException, InterruptedException { // 读取配置 Configuration conf = context.getConfiguration(); caseSensitive = conf.getBoolean("wordcount.case.sensitive", true); super.setup(context); } ? @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { if (caseSensitive) { // 是否大小写敏感 word.set(itr.nextToken()); } else { word.set(itr.nextToken().toLowerCase()); } context.write(word, one); context.getCounter(Counters.INPUT_WORDS).increment(1); } } } ? public static class CountReducer extends Reducer { ? @Override protected void reduce(Text text, Iterable values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } context.write(text, new IntWritable(sum)); } ? } ? @Override public int run(String[] args) throws Exception { Configuration conf = new Configuration(getConf()); Job job = Job.getInstance(conf, "Example Hadoop WordCount"); job.setJarByClass(WordCount.class); job.setMapperClass(CountMapper.class); job.setCombinerClass(CountReducer.class); job.setReducerClass(CountReducer.class); ? job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); ? List other_args = new ArrayList(); for (int i = 0; i < args.length; ++i) { other_args.add(args[i]); } ? FileInputFormat.addInputPath(job, new Path(other_args.get(0))); FileOutputFormat.setOutputPath(job, new Path(other_args.get(1))); int ret = job.waitForCompletion(true) ? 0 : 1; ? long inputWord = job.getCounters().findCounter(Counters.INPUT_WORDS) .getValue(); System.out.println("INPUT_WORDS:" + inputWord); logger.info("test log: " + inputWord); return ret; } ? public static void main(String[] args) throws Exception { int res = ToolRunner.run(new Configuration(), new WordCount(), args); ? System.exit(res); } ? }
运行
在eclipse导出jar包,执行以下命令
hadoop jar wordcount.jar com.fatkun.WordCount -Dwordcount.case.sensitive=false /user/fatkun/input /user/fatkun/output
参考
http://cxwangyi.blogspot.com/2009/12/wordcount-tutorial-for-hadoop-0201.html
http://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Example%3A+WordCount+v2.0
原文地址:hadoop wordcount新API例子, 感谢原作者分享。
内容总结
以上是互联网集市为您收集整理的hadoopwordcount新API例子全部内容,希望文章能够帮你解决hadoopwordcount新API例子所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错,欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 gblab@vip.qq.com 举报,一经查实,本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。