首页 / HADOOP / spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含2674字，纯文字阅读大概需要4分钟。

内容图文

spark集群启动的时候可以正常，进入 ./spark-shell 就会出现如下错误

配置文件：spark-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_51
export SCALA_HOME=/home/hadoop/scala-2.11.6

export SPARK_MASTER_IP=master24
export SPARK_MASTER_PORT=17077
export SPARK_MASTER_WEBUI_PORT=18080

export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=30g
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_INSTANCES=1INFO SparkEnv: Registering BlockManagerMaster

            错误信息如下：
15/03/2418:32:03 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150324183203-6f8e
15/03/2418:32:03 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.
15/03/2418:32:03 INFO ConnectionManager: Bound socket to port 35104 with id = ConnectionManagerId(server2,35104)
15/03/2418:32:03 INFO BlockManagerMaster: Trying to register BlockManager
15/03/2418:32:03 INFO BlockManagerInfo: Registering block manager server2:35104 with 294.9 MB RAM
15/03/2418:32:03 INFO BlockManagerMaster: Registered BlockManager
15/03/2418:32:03 INFO HttpServer: Starting HTTP Server
15/03/2418:32:03 INFO HttpBroadcast: Broadcast server started at http://192.168.1.24:4148315/03/2418:32:03 INFO HttpFileServer: HTTP File server directory is /tmp/spark-524059df-53c2-4df8-a2a0-c76c878a3d94
15/03/2418:32:03 INFO HttpServer: Starting HTTP Server
15/03/2418:32:03 INFO SparkUI: Started SparkUI at http://server12:404015/03/2418:32:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.IllegalArgumentException: java.net.UnknownHostException: cluster1
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:231)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.UnknownHostException: cluster1
... 61 more

Spark context available as sc.

scala>

 

是什么原因呢？

原文：http://www.cnblogs.com/zhanggl/p/4363671.html

内容总结

以上是互联网集市为您收集整理的spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable全部内容，希望文章能够帮你解决spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

扫描二维码推送至手机访问。

本文链接：https://qyyshop.com/info/1147934.html

来源：【匿名】