首页 > 数据库 >org.apache.spark.sql.AnalysisException: Can not create the managed table

org.apache.spark.sql.AnalysisException: Can not create the managed table

时间:2022-12-07 17:46:43浏览次数:36  
标签:managed scala AnalysisException create protocol sql apache org spark

spark执行过程中偶发性出现错误。

Traceback (most recent call last):
  File "/dfs/data9/nm-local-dir/usercache/hadoop/appcache/application_1666879209698_29104/container_e26_1666879209698_29104_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/dfs/data9/nm-local-dir/usercache/hadoop/appcache/application_1666879209698_29104/container_e26_1666879209698_29104_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o193.saveAsTable.
: org.apache.spark.sql.AnalysisException: Can not create the managed table('`xy_fx`.`mdl_uniswapv3_stat_protocol_hour_daytmp01`'). The associated location('hdfs://ns1/user/hive/warehouse/xy_fx.db/mdl_uniswapv3_stat_protocol_hour_daytmp01') already exists.;
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.validateTableLocation(SessionCatalog.scala:336)
	at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:170)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

最终找到问题原因:表已经删除,但是hdfs目录仍然存在,所以导致以上的报错。
解决方法:spark增加以下配置参数

.set("spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation","true")

标签:managed,scala,AnalysisException,create,protocol,sql,apache,org,spark
From: https://www.cnblogs.com/30go/p/16963798.html

相关文章