首页 / PYTHON / spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”

spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含11323字，纯文字阅读大概需要17分钟。

内容图文

spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”

我在Databricks上使用pySpark 2.1.

我编写了一个UDF来为pyspark数据帧的每一行生成一个唯一的uuid.我正在使用的数据帧相对较小< 10,000行.永远不应该超越那个. 我知道有内置函数spark函数zipWithIndex()和zipWithUniqueId()来生成行索引,但我已经被特别要求使用uuid来实现这个特定的项目. UDF udf_insert_uuid在小数据集上工作正常,但似乎与内置的spark函数减法冲突. 是什么导致了这个错误：

package.TreeNodeException: Binding attribute, tree: pythonUDF0#104830

更深入的驱动程序堆栈错误,它还说：

Caused by: java.lang.RuntimeException: Couldn’t find pythonUDF0#104830

这是我在下面运行的代码：

创建一个函数来生成一组unique_ids

import pandas
from pyspark.sql.functions import *
from pyspark.sql.types import *

import uuid

#define a python function
def insert_uuid():
  user_created_uuid = str( uuid.uuid1() )
  return user_created_uuid

#register the python function for use in dataframes
udf_insert_uuid = udf(insert_uuid, StringType())

创建一个包含50个元素的数据框

import pandas
from pyspark.sql.functions import *
from pyspark.sql.types import *

list_of_numbers = range(1000,1050)

temp_pandasDF = pandas.DataFrame(list_of_numbers, index=None)

sparkDF = (
  spark
  .createDataFrame(temp_pandasDF, ["data_points"])
  .withColumn("labels", when( col("data_points") < 1025, "a" ).otherwise("b"))    #if "values" < 25, then "labels" = "a", else "labels" = "b"
  .repartition("labels")
)

sparkDF.createOrReplaceTempView("temp_spark_table")

#add a unique id for each row
#udf works fine in the line of code here
sparkDF = sparkDF.withColumn("id", lit( udf_insert_uuid() ))

sparkDF.show(20, False)

ssparkDF输出：

+-----------+------+------------------------------------+
|data_points|labels|id |
+-----------+------+------------------------------------+ 
|1029 |b |d3bb91e0-9cc8-11e7-9b70-00163e9986ba|
|1030 |b |d3bb95e6-9cc8-11e7-9b70-00163e9986ba|
|1035 |b |d3bb982a-9cc8-11e7-9b70-00163e9986ba|
|1036 |b |d3bb9a50-9cc8-11e7-9b70-00163e9986ba|
|1042 |b |d3bb9c6c-9cc8-11e7-9b70-00163e9986ba|
+-----------+------+------------------------------------+
only showing top 5 rows

使用与sparkDF不同的值创建另一个DF

list_of_numbers = range(1025,1075)

temp_pandasDF = pandas.DataFrame(list_of_numbers, index=None)

new_DF = (
  spark
  .createDataFrame(temp_pandasDF, ["data_points"])
  .withColumn("labels", when( col("data_points") < 1025, "a" ).otherwise("b"))    #if "values" < 25, then "labels" = "a", else "labels" = "b"
  .repartition("labels"))

new_DF.show(5, False)

new_DF输出：

+-----------+------+
|data_points|labels|
+-----------+------+
|1029 |b |
|1030 |b |
|1035 |b |
|1036 |b |
|1042 |b | 
+-----------+------+
only showing top 5 rows

将new_DF中的值与spark_DF进行比较

values_not_in_new_DF = (new_DF.subtract(sparkDF.drop("id")))

将uuid添加到udf的每一行并显示它

display(values_not_in_new_DF
       .withColumn("id", lit( udf_insert_uuid()))   #add a column of unique uuid's
       )

以下错误结果：

package.TreeNodeException: Binding attribute, tree: pythonUDF0#104830
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: pythonUDF0#104830 at
org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56) at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:88) at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:87) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:268) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:268) at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:267) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:273) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:273) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) at
org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:273) at
org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:257) at
org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:87) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$33.apply(HashAggregateExec.scala:473) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$33.apply(HashAggregateExec.scala:472) at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at
scala.collection.AbstractTraversable.map(Traversable.scala:105) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec.generateResultCode(HashAggregateExec.scala:472) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec.doProduceWithKeys(HashAggregateExec.scala:610) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec.doProduce(HashAggregateExec.scala:148) at
org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83) at
org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:78) at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at
org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:78) at
org.apache.spark.sql.execution.aggregate.HashAggregateExec.produce(HashAggregateExec.scala:38) at
org.apache.spark.sql.execution.WholeStageCodegenExec.doCodeGen(WholeStageCodegenExec.scala:313) at
org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:354) at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at
org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at
org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225) at
org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308) at
org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at
org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:2807) at
org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2132) at
org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2132) at
org.apache.spark.sql.Dataset$$anonfun$60.apply(Dataset.scala:2791) at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:87) at
org.apache.spark.sql.execution.SQLExecution$.withFileAccessAudit(SQLExecution.scala:53) at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:70) at
org.apache.spark.sql.Dataset.withAction(Dataset.scala:2790) at
org.apache.spark.sql.Dataset.head(Dataset.scala:2132) at
org.apache.spark.sql.Dataset.take(Dataset.scala:2345) at
com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation0(OutputAggregator.scala:81) at
com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation(OutputAggregator.scala:42) at
com.databricks.backend.daemon.driver.PythonDriverLocal$$anonfun$getResultBuffer$1.apply(PythonDriverLocal.scala:461) at
com.databricks.backend.daemon.driver.PythonDriverLocal$$anonfun$getResultBuffer$1.apply(PythonDriverLocal.scala:441) at
com.databricks.backend.daemon.driver.PythonDriverLocal.withInterpLock(PythonDriverLocal.scala:394) at
com.databricks.backend.daemon.driver.PythonDriverLocal.getResultBuffer(PythonDriverLocal.scala:441) at
com.databricks.backend.daemon.driver.PythonDriverLocal.com$databricks$backend$daemon$driver$PythonDriverLocal$$outputSuccess(PythonDriverLocal.scala:428) at
com.databricks.backend.daemon.driver.PythonDriverLocal$$anonfun$repl$3.apply(PythonDriverLocal.scala:178) at
com.databricks.backend.daemon.driver.PythonDriverLocal$$anonfun$repl$3.apply(PythonDriverLocal.scala:175) at
com.databricks.backend.daemon.driver.PythonDriverLocal.withInterpLock(PythonDriverLocal.scala:394) at
com.databricks.backend.daemon.driver.PythonDriverLocal.repl(PythonDriverLocal.scala:175) at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$2.apply(DriverLocal.scala:230) at
com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$2.apply(DriverLocal.scala:211) at
com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:173) at
scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at
com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:168) at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:39) at
com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:206) at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:39) at
com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:211) at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589) at
com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:589) at
scala.util.Try$.apply(Try.scala:161) at
com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:584) at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:488) at
com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:391) at
com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:348) at
com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:215) at
java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Couldn’t find pythonUDF0#104830 in [data_points#104799L,labels#104802] at
scala.sys.package$.error(package.scala:27) at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:94) at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:88) at
org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52) … 82 more

解决方法:

我在运行脚本时遇到了与您相同的错误.我发现使其工作的唯一方法是将UDF传递给一列而不是没有参数：

def insert_uuid(col):
    user_created_uuid = str( uuid.uuid1() )
    return user_created_uuid
udf_insert_uuid = udf(insert_uuid, StringType())

然后在标签上调用它,例如：

values_not_in_new_DF\
    .withColumn("id", udf_insert_uuid("labels"))\
    .show()

无需使用点燃

内容总结

以上是互联网集市为您收集整理的spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”全部内容，希望文章能够帮你解决spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”所遇到的程序开发问题。如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容，请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

扫描二维码推送至手机访问。

本文链接：https://qyyshop.com/info/781106.html

来源：【匿名】

【上一篇】找到多个重叠矩形的并集 – OpenCV python 【下一篇】浅谈PHP运行Python脚本的方法

更多 ►

【spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”】教程文章相关的互联网学习教程文章

python 中exception，class学习

python 中exception，class 学习instroduction:Object2 = Object1 ( like java)if Object1 is class object , then copy by reference;if Object1 is basic type, then copy by value1. exception主要结构：try:exception ValueError:exception ZeroDivisionError:exception NameError:exception TypeError:exception:finally:2. custom exceptionclass Error(Exception): passdef MyError(Error): def __init__(self,value): pri...

Python2和Python3中raise Exception【图】

今天写了个记录错误日志的脚本，发现Python3中的raise Exception发生了变化，如下：错误搜了下才知道原来是Python3.6已经不支持这种写法了，所以用了心的方式，如下：正确原文：https://www.cnblogs.com/April-Chou-HelloWorld/p/8836265.html

python handle exception【代码】

1. handle exceptionimport systry:a=1/1 except Exception, e:print"failed", sys.exc_info()[0] else:print"no exception"finally:print"execute final"2. print exceptiontry:raise Exception("aaa","bbb")#a=1/0except Exception as e :print (type(e))print(e.args)print (e) 原文：http://www.cnblogs.com/phoenix13suns/p/3978929.html

关于Python中异常(Exception)

这篇文章介绍的内容是关于关于Python中异常(Exception) ，有着一定的参考价值，现在分享给大家，有需要的朋友可以参考一下异常是指程序中的例外，违例情况。异常机制是指程序出现错误后，程序的处理方法。当出现错误后，程序的执行流程发生改变，程序的控制权转移到异常处理。下面这篇文章主要汇总了关于Python中异常(Exception)的相关资料，需要的朋友可以参考下。前言Exception类是常用的异常类，该类包括StandardError，StopIte...

详解Python中Exception异常处理

关于异常处理：Python中类定义可以作为对象传递，例子中类B继承了Exception类，在循环中每次raise一个异常类对象B，Except抓到这个异常，循环了3次，所以输出了B B B。例子：class B(Exception): passclass C(B): passclass D(C): passfor cls in [B, C, D]: try: raise cls() except B: print("B") except C: print("C") except D: print("D")执行结果：BBB更多详解Python中Exc...

关于Python中异常(Exception)的汇总

异常是指程序中的例外，违例情况。异常机制是指程序出现错误后，程序的处理方法。当出现错误后，程序的执行流程发生改变，程序的控制权转移到异常处理。下面这篇文章主要汇总了关于Python中异常(Exception)的相关资料，需要的朋友可以参考下。前言Exception类是常用的异常类，该类包括StandardError，StopIteration, GeneratorExit, Warning等异常类。python中的异常使用继承结构创建，可以在异常处理程序中捕获基类异常，也可以捕...

异常(exception)和执行失败有什么区别？

例如一个User Class 的 add 方法，在成功的情况下返回用户对象实例，在失败的情况返回False并可以通过getError方法获取失败原因字符串........说到这里，我好像明白了，难道add方法总是应该返回用户对象，否则抛出异常吗？但是这样的话，他们的代码量没什么区别的啊。问题在于即使调用add方法处没有捕捉异常，该异常也能进一步向上抛出直至被处理或引发进程崩溃？可是说到底，这和程序自然崩溃有什么区别呢？---- 以上为自言自语，...

安装 opencv-python【报错】--【ERROR: Exception: Traceback (most recent call last):】【代码】【图】

使用 pip install opencv-python 命令安装报错：解决方案 1. pip install -i https://pypi.tuna.tsinghua.edu.cn/simple opencv-python我依旧报错！！！！ 2 pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn opencv-python我依旧报错！！！ 3 解决啦使用豆瓣的链接 pip install opencv-python -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com成功安装！...

python -m onnxsim 报错：RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION ，UpsampleMode) con【代码】

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /Users/runner/work/1/s/onnxruntime/core/providers/cpu/tensor/upsample.h:271 void onnxruntime::UpsampleBase::ScalesValidation(const std::vector<float> &, const onnxruntime::UpsampleMode) const scale >= 1 was false. Scale value should be greater than or equal to 1.原因是在 torch.onnx.export 的时候没有配置...

Python中使用selenium出现Exception has occurred: WebDriverException【代码】

引言最近自己在开始弄关于Python Spider的内容，就像大家一开始学习一样，都是默默地从环境配置开始。总得来说，这次关于Python Spider的环境配置还是挺快的，基本上没有问题，主要是在这之前就把一些必要的软件给安装好了，例如anaconda，Pycharm，Vscode这些，之后便是导入相关的包，例如selenium和requests这些。如果要学python spider的话，我觉得一般会下一个chromedirver作为工具吧，反正我是这样子做的。下完之后，我自己...

Python - Django - 中间件 process_exception【图】

process_exception(self, request, exception) 函数有两个参数，exception 是视图函数异常产生的 Exception 对象 process_exception 函数的执行顺序是按照 settings.py 中设置的中间件的顺序的倒序执行 process_exception 函数只在视图函数中出现异常的时候才执行，它返回的值可以是 None，也可以是一个 HttpResponse 对象如果返回 None，则继续由下一个中间件的 process_exception 方法来处理异常如果返回 HttpResponse，将调用...

【python】Exception in thread Thread-1:Traceback (most recent call last)

原文链接：https://blog.csdn.net/a519640026/article/details/76157930用多线程和队列解决生产者、消费者问题时程序报错：Exception in thread Thread-1:Traceback (most recent call last)。原因：由于在python 的函数参数中没有加入self导致的。参考例子：|Python 多线程|Queue队列|生产者消费者模式|#!/usr/bin/env python # -*- coding: utf-8 -*-import threading import Queue import random import timeclass Producter(...

返回Exception实例而不是在Python中提升它的缺点是什么？【代码】

我一直在用python-couchdb和desktopcouch做一些工作.在我提交的其中一个补丁中,我从couchdb中包装了db.update函数.对于任何不熟悉python-couchdb的人来说,功能如下：def update(self, documents, **options):"""Perform a bulk update or insertion of the given documents using asingle HTTP request.>>> server = Server('http://localhost:5984/')>>> db = server.create('python-tests')>>> for doc in db.update([... Do...

python异常(高级) Exception【图】

异常(高级) Exception异常回顾:try-except 语句捕获(接收)异常通知,把异常流程变为正常流程try-finally 语句执行必须要执行的语句.raise 语句发送异常通知,同时进入异常流程assert 语句发送AssertionError异常with 语句 with语句语法:with 表达式1 [as 变量1], 表达式2 [as 变量2], ...:语句块作用:使用于对资源进行访问的场合,确保使用过程中不管是否发生异常都会说明:with语句同try-finally语句一样,不会改变程序的状态(异...

python – 为什么我得到了“Exception：(404,u’Not Found’)”和Suds【代码】

我正在尝试使用Suds连接到SugarCRM soap服务(什么是正确的术语？)：from suds.client import Clienturl = "http://localhost/sugarcrm/soap.php?wsdl" client = Client(url) session = client.service.login("usr", "pwd")但是最后一行抛出异常：ERROR:suds.client:<?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:ns3="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/enc...

首页 / PYTHON / spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”

spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”

内容导读

内容图文

内容总结

内容备注

内容手机端

【spark“package.TreeNodeException”错误python“java.lang.RuntimeException：找不到pythonUDF”】教程文章相关的互联网学习教程文章

TREENODE - 相关标签

RUNTIME - 相关标签

PYTHON - 技术教程分类

PYTHON - 最新教程

PYTHON - 最热教程