首页 / 算法 / 运行Mahout分类算法，分析20newsgroup的分类过程

运行Mahout分类算法，分析20newsgroup的分类过程

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了运行Mahout分类算法，分析20newsgroup的分类过程，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含11767字，纯文字阅读大概需要17分钟。

内容图文

1.由于虚拟机不能联网，直接运行./example/bin/classify-20newsgroups.sh出现很多问题，索性直接看目标运行classify-20newsgroups.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# Downloads the 20newsgroups dataset, trains and tests a classifier.
#
# To run:  change into the mahout directory and type:
# examples/bin/classify-20newsgroups.sh

if [ "$1" = "--help" ] || [ "$1" = "--?" ]; then
  echo "This script runs SGD and Bayes classifiers over the classic 20 News Groups."
  exit
fi

SCRIPT_PATH=${0%/*}
if [ "$0" != "$SCRIPT_PATH" ] && [ "$SCRIPT_PATH" != "" ]; then
  cd $SCRIPT_PATH
fi
START_PATH=`pwd`

if [ "$HADOOP_HOME" != "" ] && [ "$MAHOUT_LOCAL" == "" ] ; then
  HADOOP="$HADOOP_HOME/bin/hadoop"
  if [ ! -e $HADOOP ]; then
    echo "Can‘t find hadoop in $HADOOP, exiting"
    exit 1
  fi
fi

WORK_DIR=/tmp/mahout-work-${USER}
algorithm=( cnaivebayes naivebayes sgd clean)
if [ -n "$1" ]; then
  choice=$1
else
  echo "Please select a number to choose the corresponding task to run"
  echo "1. ${algorithm[0]}"
  echo "2. ${algorithm[1]}"
  echo "3. ${algorithm[2]}"
  echo "4. ${algorithm[3]} -- cleans up the work area in $WORK_DIR"
  read -p "Enter your choice : " choice
fi

echo "ok. You chose $choice and we‘ll use ${algorithm[$choice-1]}"
alg=${algorithm[$choice-1]}

if [ "x$alg" != "xclean" ]; then
  echo "creating work directory at ${WORK_DIR}"

  mkdir -p ${WORK_DIR}
  if [ ! -e ${WORK_DIR}/20news-bayesinput ]; then
    if [ ! -e ${WORK_DIR}/20news-bydate ]; then
      if [ ! -f ${WORK_DIR}/20news-bydate.tar.gz ]; then
        echo "Downloading 20news-bydate"
        curl http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz -o ${WORK_DIR}/20news-bydate.tar.gz
      fi
      mkdir -p ${WORK_DIR}/20news-bydate
      echo "Extracting..."
      cd ${WORK_DIR}/20news-bydate && tar xzf ../20news-bydate.tar.gz && cd .. && cd ..
    fi
  fi
fi
#echo $START_PATH
cd $START_PATH
cd ../..

set -e

if [ "x$alg" == "xnaivebayes"  -o  "x$alg" == "xcnaivebayes" ]; then
  c=""

  if [ "x$alg" == "xcnaivebayes" ]; then
    c=" -c"
  fi

  set -x
  echo "Preparing 20newsgroups data"
  rm -rf ${WORK_DIR}/20news-all
  mkdir ${WORK_DIR}/20news-all
  cp -R ${WORK_DIR}/20news-bydate/*/* ${WORK_DIR}/20news-all

  if [ "$HADOOP_HOME" != "" ] && [ "$MAHOUT_LOCAL" == "" ] ; then
    echo "Copying 20newsgroups data to HDFS"
    set +e
    $HADOOP dfs -rmr ${WORK_DIR}/20news-all
    set -e
    $HADOOP dfs -put ${WORK_DIR}/20news-all ${WORK_DIR}/20news-all
  fi

  echo "Creating sequence files from 20newsgroups data"
  ./bin/mahout seqdirectory     -i ${WORK_DIR}/20news-all     -o ${WORK_DIR}/20news-seq -ow

  echo "Converting sequence files to vectors"
  ./bin/mahout seq2sparse     -i ${WORK_DIR}/20news-seq     -o ${WORK_DIR}/20news-vectors  -lnorm -nv  -wt tfidf

  echo "Creating training and holdout set with a random 80-20 split of the generated vector dataset"
  ./bin/mahout split     -i ${WORK_DIR}/20news-vectors/tfidf-vectors     --trainingOutput ${WORK_DIR}/20news-train-vectors     --testOutput ${WORK_DIR}/20news-test-vectors      --randomSelectionPct 40 --overwrite --sequenceFiles -xm sequential

  echo "Training Naive Bayes model"
  ./bin/mahout trainnb     -i ${WORK_DIR}/20news-train-vectors -el     -o ${WORK_DIR}/model     -li ${WORK_DIR}/labelindex     -ow $c

  echo "Self testing on training set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-train-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c

  echo "Testing on holdout set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-test-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c

elif [ "x$alg" == "xsgd" ]; then
  if [ ! -e "/tmp/news-group.model" ]; then
    echo "Training on ${WORK_DIR}/20news-bydate/20news-bydate-train/"
    ./bin/mahout org.apache.mahout.classifier.sgd.TrainNewsGroups ${WORK_DIR}/20news-bydate/20news-bydate-train/
  fi
  echo "Testing on ${WORK_DIR}/20news-bydate/20news-bydate-test/ with model: /tmp/news-group.model"
  ./bin/mahout org.apache.mahout.classifier.sgd.TestNewsGroups --input ${WORK_DIR}/20news-bydate/20news-bydate-test/ --model /tmp/news-group.model
elif [ "x$alg" == "xclean" ]; then
  rm -rf ${WORK_DIR}
  rm -rf /tmp/news-group.model
fi
# Remove the work directory
#

上边是Mahout自带的classify-20newsgroups.sh文件的内容，下边是改之后的内容。

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# Downloads the 20newsgroups dataset, trains and tests a classifier.
#
# To run:  change into the mahout directory and type:
# examples/bin/classify-20newsgroups.sh

if [ "$1" = "--help" ] || [ "$1" = "--?" ]; then
  echo "This script runs SGD and Bayes classifiers over the classic 20 News Groups."
  exit
fi

SCRIPT_PATH=${0%/*}
if [ "$0" != "$SCRIPT_PATH" ] && [ "$SCRIPT_PATH" != "" ]; then
  cd $SCRIPT_PATH
fi
START_PATH=`pwd`

if [ "$HADOOP_HOME" != "" ] && [ "$MAHOUT_LOCAL" == "" ] ; then
  HADOOP="$HADOOP_HOME/bin/hadoop"
  if [ ! -e $HADOOP ]; then
    echo "Can‘t find hadoop in $HADOOP, exiting"
    exit 1
  fi
fi
//工作目录
WORK_DIR=/tmp/mahout-work-${USER}
algorithm=( cnaivebayes naivebayes sgd clean)
if [ -n "$1" ]; then
  choice=$1
else
  echo "Please select a number to choose the corresponding task to run"
  echo "1. ${algorithm[0]}"
  echo "2. ${algorithm[1]}"
  echo "3. ${algorithm[2]}"
  echo "4. ${algorithm[3]} -- cleans up the work area in $WORK_DIR"
  read -p "Enter your choice : " choice
fi

echo "ok. You chose $choice and we‘ll use ${algorithm[$choice-1]}"
alg=${algorithm[$choice-1]}
//然后从http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz直接下载，将下载的文件放到WORK_DIR中然后
if [ "x$alg" != "xclean" ]; then
  echo "creating work directory at ${WORK_DIR}"
  mkdir -p ${WORK_DIR}
  if [ ! -e ${WORK_DIR}/20news-bayesinput ]; then
    if [ ! -e ${WORK_DIR}/20news-bydate ]; then
      if [ ! -f ${WORK_DIR}/20news-bydate.tar.gz ]; then
       echo "Downloading 20news-bydate"
        curl http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz -o ${WORK_DIR}/20news-bydate.tar.gz
      fi
      mkdir -p ${WORK_DIR}/20news-bydate
      echo "Extracting..."
      cd ${WORK_DIR}/20news-bydate && tar xzf ../20news-bydate.tar.gz && cd .. && cd ..
    fi
  fi
fi

#echo $START_PATH
cd $START_PATH
cd ../..

set -e

if [ "x$alg" == "xnaivebayes"  -o  "x$alg" == "xcnaivebayes" ]; then
  c=""

  if [ "x$alg" == "xcnaivebayes" ]; then
    c=" -c"
  fi
  set -x
  echo "Preparing 20newsgroups data"
  rm -rf ${WORK_DIR}/20news-all
  mkdir ${WORK_DIR}/20news-all
  cp -R ${WORK_DIR}/20news-bydate/*/* ${WORK_DIR}/20news-all        
//20news-bydate中包括训练集和测试集，合起来是完整的20newsgroups //这3句是自己加上的，直接运行会找不到文件问题出现。
  export HADOOP=hadoop  #I add it
  export HWORK_DIR=hdfs://localhost:9000${WORK_DIR} 
  export CLASSPATH=$CLASSPATH:$MAHOUT_HOME/lib
#  if [ "$HADOOP_HOME" != "" ] && [ "$MAHOUT_LOCAL" == "" ] ; then
    echo "Copying 20newsgroups data to HDFS"
#    set +e
#    $HADOOP dfs -rmr ${WORK_DIR}/20news-all
#    set -e
#    $HADOOP dfs -put ${WORK_DIR}/20news-all ${HWORK_DIR}/20news-all
# fi

  echo "Creating sequence files from 20newsgroups data"
  ./bin/mahout seqdirectory     -i ${WORK_DIR}/20news-all     -o ${WORK_DIR}/20news-seq -ow

  echo "Converting sequence files to vectors"
  ./bin/mahout seq2sparse     -i ${WORK_DIR}/20news-seq     -o ${WORK_DIR}/20news-vectors  -lnorm -nv  -wt tfidf

  echo "Creating training and holdout set with a random 80-20 split of the generated vector dataset"
  ./bin/mahout split     -i ${WORK_DIR}/20news-vectors/tfidf-vectors     --trainingOutput ${WORK_DIR}/20news-train-vectors     --testOutput ${WORK_DIR}/20news-test-vectors      --randomSelectionPct 40 --overwrite --sequenceFiles -xm sequential

  echo "Training Naive Bayes model"
  ./bin/mahout trainnb     -i ${WORK_DIR}/20news-train-vectors -el     -o ${WORK_DIR}/model     -li ${WORK_DIR}/labelindex     -ow $c

  echo "Self testing on training set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-train-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c

  echo "Testing on holdout set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-test-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c

elif [ "x$alg" == "xsgd" ]; then
  if [ ! -e "/tmp/news-group.model" ]; then
    echo "Training on ${WORK_DIR}/20news-bydate/20news-bydate-train/"
    ./bin/mahout org.apache.mahout.classifier.sgd.TrainNewsGroups ${WORK_DIR}/20news-bydate/20news-bydate-train/
  fi
  echo "Testing on ${WORK_DIR}/20news-bydate/20news-bydate-test/ with model: /tmp/news-group.model"
  ./bin/mahout org.apache.mahout.classifier.sgd.TestNewsGroups --input ${WORK_DIR}/20news-bydate/20news-bydate-test/ --model /tmp/news-group.model
elif [ "x$alg" == "xclean" ]; then
  rm -rf ${WORK_DIR}
  rm -rf /tmp/news-group.model
fi


2.


  echo "Creating sequence files from 20newsgroups data"
  ./bin/mahout seqdirectory     -i ${WORK_DIR}/20news-all     -o ${WORK_DIR}/20news-seq -ow


从MAHOUT_HOME/conf文件夹下的driver.classes.default.props文件可以找到，此命令实际是运行org.apache.mahout.text.SequenceFilesFromDirectory类，此类是一个Hadoop的Job。只有Mapper，SequenceFilesFromDirectoryMapper，没有Reducer。

3.
echo "Converting sequence files to vectors"
  ./bin/mahout seq2sparse     -i ${WORK_DIR}/20news-seq     -o ${WORK_DIR}/20news-vectors  -lnorm -nv  -wt tfidf
实际运行org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles类，包含4个Job DocumentProcessor，DictionaryVectorizer，HighDFWordsPruner和TFIDFConverter

4.
  echo "Creating training and holdout set with a random 80-20 split of the generated vector dataset"
  ./bin/mahout split     -i ${WORK_DIR}/20news-vectors/tfidf-vectors     --trainingOutput ${WORK_DIR}/20news-train-vectors     --testOutput ${WORK_DIR}/20news-test-vectors      --randomSelectionPct 40 --overwrite --sequenceFiles -xm sequential


实际运行org.apache.mahout.utils.SplitInput类，Job类是org.apache.mahout.utils.SplitInputJob。SplitInputMapper，SplitInputReducer。

5.
  echo "Training Naive Bayes model"
  ./bin/mahout trainnb     -i ${WORK_DIR}/20news-train-vectors -el     -o ${WORK_DIR}/model     -li ${WORK_DIR}/labelindex     -ow $c


实际运行org.apache.mahout.classifier.naivebayes.training.TrainNaiveBayesJob类，3步，indexInstances，weightSummer和thetaSummer。

6.
 echo "Self testing on training set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-train-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c实际运行org.apache.mahout.classifier.naivebayes.test.TestNaiveBayesDriver类，BayesTestMapper。

7.
  echo "Testing on holdout set"

  ./bin/mahout testnb     -i ${WORK_DIR}/20news-test-vectors    -m ${WORK_DIR}/model     -l ${WORK_DIR}/labelindex     -ow -o ${WORK_DIR}/20news-testing $c



原文：http://blog.csdn.net/xichenguan/article/details/23029835

内容总结

以上是互联网集市为您收集整理的运行Mahout分类算法，分析20newsgroup的分类过程全部内容，希望文章能够帮你解决运行Mahout分类算法，分析20newsgroup的分类过程所遇到的程序开发问题。如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容，请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

扫描二维码推送至手机访问。

本文链接：https://qyyshop.com/info/1161899.html

来源：【匿名】

【上一篇】算法的时间复杂度O 【下一篇】浅谈php实现映射的两种方法（链表和二叉树）

更多 ►

【运行Mahout分类算法，分析20newsgroup的分类过程】教程文章相关的互联网学习教程文章

数据结构与算法分析

线性表数据元素的排列方式是线性的顺序表：顺序表是在计算机中以数组的形式保存的线性表结构除了头尾，其他的元素依次首尾相连在内存中是一块连续的存储空间，每个元素占用相同的空间，所以顺序表支持随机访问e[i]=e[1]+e[i-1]*length, 1<=i<=n,length是单个元素所占的空间Java数组 Java在定义数组的时候，在堆里面分配一个连续的固定大小的空间，用于存放基本数据类型或者对象的引用。顺序表支持随机存取，所以Java数组用任意下...

JVM之可达性分析算法和引用

1、可达性分析算法　通过一系列称为GC Roots的根对象作为起始节点集，根据引用关系向下搜索，搜索走过的路径叫做引用链，如果某个对象到GC Roots节点集没有任何的引用链也被称为不可达，则证明这个对象不可能再被使用。　可作为GC Roots根节点的：在虚拟机栈（栈帧中的本地变量表）中引用的对象，譬如各个线程被调用的方法堆栈中使用到的参数、局部变量、临时变量等在方法区中类静态属性引用的对象，譬如Java类的引用类型静态变量...

Java中使用TreeMap权重随机算法，以及验证与分析【代码】

权重下随机，就是给定各个值不同的权重，再根据权重的比例随机选出一个值 1/** 2 * Created by Jungle on 2020/2/23.3 *4 * @author JungleZhang5 * @version 1.0.06 * @Description 权重下随机的算法7*/ 8publicclass WeightRandom<K, V extends Number> {9private TreeMap<Double, K> weightMap = new TreeMap<>(); 1011public WeightRandom(@NotNull List<Pair<K, V>> list) { 12// 先排除权重为0的项13 Iterator<Pair<...

《数据结构与算法分析：C语言描述》复习——第六章“排序”——选择排序【代码】

2014.06.17 01:17简介：　　选择排序是一种O(n^2)级别的交换排序算法，属于新手必学算法。描述：　　个人觉得选择排序的代码是所有排序中最直观，最符合人类大脑思维的了。当我第一次有排序的需求时（初中时自学了一点C语言，算是人生第一次写代码），我自己试着写下的代码就是选择排序，当然我上了大学才知道“选择排序”是什么。很显然，直观且容易实现的算法基本都是最鹾的，而不直观且容易实现的算法基本都是最神的。很显然，选...

算法设计与分析(李春保)练习题答案v2

----------------------------------------------------- Page 1 -----------------------------------------------------1.1 第 1 章 ─ 概论 1.1.1 练习题 1. 下列关于算法的说法中正确的有（）。 Ⅰ . 求解某一类问题的算法是唯一的 Ⅱ . 算法必须在有限步操作之后停止 Ⅲ . 算法的每一步操作必须是明确的，不能有歧义或含义模糊 Ⅳ . 算法执行后一...

『嗨威说』算法设计与分析 - 算法第二章上机实践报告（二分查找 / 改写二分搜索算法 / 两个有序序列的中位数）【代码】【图】

本文索引目录：一、PTA实验报告题1 ：二分查找　　1.1　　实践题目　　1.2　　问题描述　　1.3　　算法描述　　1.4　　算法时间及空间复杂度分析二、PTA实验报告题2 ：改写二分搜索算法　　2.1　　实践题目　　2.2　　问题描述　　2.3　　算法描述　　2.4　　算法时间及空间复杂度分析三、PTA实验报告题3 ：两个有序序列的中位数　　3.1　　实践题目　　3.2　　问题描述　　3.3　　算法描述　　3.4　　算法时间及空间复杂度分析...

Pixhawk之姿态控制篇（1）_源码算法分析（超级有料）【图】

数据结构与算法之美专栏学习笔记-复杂度分析【代码】

复杂度分析什么是复杂度分析数据结构和算法解决是“如何让计算机更快时间、更省空间的解决问题”。因此需从执行时间和占用空间两个维度来评估数据结构和算法的性能。分别用时间复杂度和空间复杂度两个概念来描述性能问题，二者统称为复杂度。复杂度描述的是算法执行时间（或占用空间）与数据规模的增长关系。为什么要进行复杂度分析和性能测试相比，复杂度分析有不依赖执行环境、成本低、效率高、易操作、指导性强的特点。掌握复杂...

算法效率分析【图】

希望大家还是在实践当中感受效率，理解效率，写出高质量的代码和算法，仅仅作为参考，不要误导大家。排版尽量舒服吧，尽力了。一般而言分析算法效率的方式有两种，即：时间效率和空间效率。时间效率也称为时间复杂度；空间效率也称为空间复杂度。在计算机技术发展的几十年中，空间资源变得不是非常重要了，因此在一般的算法分析中，讨论的主要是时间复杂度，当然空间复杂度的分析也是如此。在算法分析中，我们不使用时间的标准单位...

寒假 8 （算法分析和递归，一些例子）

数据结构书前两章。附：递归的四个法则base cases；making progress；design rule; compound interest rule 算法里面的大小o不是高阶无穷的意思，就是说当n很大时，a比b小，也就是说，a的增长率小只定义了小，大，相等，严格小，用到的一般只有小有些递归和for循环等价，有些递归不可以改写成for循环算法分析，时间——语句——从最深层处开始计算，函数调用要展开，递归要展开计算任何事情不要超过一次：不要重复。 max sub...

10种排序算法分析【代码】【图】

10种排序算法，分别是直接插入排序，折半插入排序，希尔排序，冒泡排序，快速排序，直接选择排序，树形排序，堆排序，归并排序，基数排序。各有千秋，但依旧有优劣之分，熟悉每一个算法，对于我们的代码优化，也将事半功倍。 1，直接插入排序：基本思想：假设待排的n个记录存放在变量R中，首先将R[1]看做是有序区，将后n - 1个数组元素看作是无序区；然后将无序区的第一个元素R[2]插入到前面有序区的适当位置，从而得到新的有序区R...

3.4 网页分析算法

在搜索引擎中，爬虫爬取了对应的网页之后，会将网页存储到服务器的原始数据库中，之后搜索引擎会对这些网页进行分析并确定各网页的重要性，即会影响用户的检索的排名结果。对于这些重要性的确定及排名结果的确定需要算法来解决，所以先来了解一下算法。搜索引擎的网页分析算法主要分为3类：基于用户行为的网页分析算法、基于网络拓扑的网页分析算法、基于网页内容的网页分析算法。接下来我们分别对这些算法进行讲解。搜索引擎的网页...

Python 实现关联规则分析Apriori算法【代码】【图】

# -*- coding:utf-8 -*-import sysreload(sys) sys.setdefaultencoding("utf8")def load_data_set():data_set = [[‘beer‘, ‘baby diapers‘, ‘shorts‘], [‘baby diapers‘, ‘shorts‘], [‘baby diapers‘, ‘milk‘], [‘beer‘, ‘baby diapers‘, ‘shorts‘], [‘beer‘, ‘milk‘], [‘baby diapers‘, ‘milk‘], [‘beer‘, ‘milk‘], [‘beer‘, ‘baby diapers‘, ‘milk‘, ‘shorts‘], [‘beer‘, ‘baby diap...