MapReduce学习总结之java版wordcount实现
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了MapReduce学习总结之java版wordcount实现,小编现在分享给大家,供广大互联网技能从业者学习和参考。文章包含2840字,纯文字阅读大概需要5分钟。
内容图文
![MapReduce学习总结之java版wordcount实现](/upload/InfoBanner/zyjiaocheng/834/4d1b2581ea65424392b9d261a529db22.jpg)
一、代码实现:
package rdb.com.hadoop01.mapreduce; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; /** * mapreduce word count 应用 * @author rdb * */ public class WordCountApp { /** * map读取输入文件 * @author rdb * */ public static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable>{ LongWritable one = new LongWritable(1); @Override protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, LongWritable>.Context context) throws IOException, InterruptedException { //接收每一行数据 String line = value.toString(); //按空格进行分割 String[] words = line.split(" "); for(String word :words){ //通过上下文把map处理结果输出 context.write(new Text(word), one); } } } /** * reduce程序,归并统计 * @author rdb * */ public static class MyReduce extends Reducer<Text, LongWritable, Text, LongWritable>{ @Override protected void reduce(Text key, Iterable<LongWritable> values, Reducer<Text, LongWritable, Text, LongWritable>.Context context) throws IOException, InterruptedException { long sum = 0; for (LongWritable value : values){ //求单词次数 sum += value.get(); } //通过上下文把reduce处理结果输出 context.write(key, new LongWritable(sum)); } } /** * 自定义driver:封装mapreduce作业所有信息 *@param args * @throws IOException */ public static void main(String[] args) throws Exception { //创建配置 Configuration configuration = new Configuration(); //清理已经存在的输出目录 Path out = new Path(args[1]); FileSystem fileSystem = FileSystem.get(configuration); if(fileSystem.exists(out)){ fileSystem.delete(out, true); System.out.println("output exists,but it has deleted"); } //创建job Job job = Job.getInstance(configuration,"WordCount"); //设置job的处理类 job.setJarByClass(WordCountApp.class); //设置作业处理的输入路径 FileInputFormat.setInputPaths(job, new Path(args[0])); //设置map相关的参数 job.setMapperClass(MyMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); //设置reduce相关参数 job.setReducerClass(MyReduce.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); //设置作业处理的输出路径 FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true)? 0 : 1) ; } }
二、运行结果:
调用:
hadoop jar ~/lib/hadoop01-0.0.1-SNAPSHOT.jar rdb.com.hadoop01.mapreduce.WordCountApp hdfs://hadoop01:8020/hello.txt hdfs://hadoop01:8020/output/wc
输入的文件内容:
Deer Bear River Car Car River Deer Car Bear
输出的结果:
Bear 2 Car 3 Deer 2 River 2
内容总结
以上是互联网集市为您收集整理的MapReduce学习总结之java版wordcount实现全部内容,希望文章能够帮你解决MapReduce学习总结之java版wordcount实现所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错,欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 gblab@vip.qq.com 举报,一经查实,本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。