spark-sql Required-field-'filesAdded'-is-unset

1. 背景

使用sparkSQL计算数据向一个已经存在数据的分区中写数据报错

使用版本：
Spark2 2.3.2
Hive 3.1.0

错误信息如下：

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Required field 'filesAdded' is unset! Struct:InsertEventRequestData(filesAdded:null);
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
    at org.apache.spark.sql.hive.HiveExternalCatalog.loadDynamicPartitions(HiveExternalCatalog.scala:871)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:205)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:115)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
    ......
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Required field 'filesAdded' is unset! Struct:InsertEventRequestData(filesAdded:null)
    at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:1949)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1876)
    ... 37 more
Caused by: org.apache.thrift.TApplicationException: Required field 'filesAdded' is unset! Struct:InsertEventRequestData(filesAdded:null)
    at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_fire_listener_event(ThriftHiveMetastore.java:4182)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.fire_listener_event(ThriftHiveMetastore.java:4169)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.fireListenerEvent(HiveMetaStoreClient.java:1954)
    at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
    at com.sun.proxy.$Proxy19.fireListenerEvent(Unknown Source)
    at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:1947)
    ... 38 more

2. 解决问题

这是hive的一个bug ， 可以修改 hive-site.xml 配置文件中的 hive.metastore.dml.events 为 false 来解决此问题。

如果修改配置没有解决还可能是下面的原因

由于Spark SQL中的查询操作无法正确连接到Hive Metastore，导致在执行INSERT INTO语句时出现异常。这可能是因为您的Hive版本与Spark SQL不兼容，或者Hive Metastore配置不正确。

• 解决办法：

检查 Hive Metastore 的配置是否正确。
确保在 Spark SQL 配置文件中指定了正确的 Hive Metastore URI，并且具有执行插入操作所需的权限
修改配置以后，重启Hive服务和Spark应用程序，以确保所有配置更改和依赖项更新都已生效。