Flink Log4j jar包冲突问题解决

exception flink
flink
发布日期:   2023-09-01

flink 任务添加了 elasticsearch 之后 log4j 日志的 jar出现了冲突， 下面是一些报错日志以及解决方式：

1. 提交任务时未成功提交的日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/bigdata/data-process/flink/flink-1.13.6/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-08-30 14:00:59,937 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/bigdata/data-process/flink/flink-1.13.6/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-08-30 14:01:00,242 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at nodej01/192.168.71.70:8050
2023-08-30 14:01:00,506 INFO  org.apache.hadoop.yarn.client.AHSProxy                       [] - Connecting to Application History server at nodej02/192.168.71.71:10200
2023-08-30 14:01:00,515 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2023-08-30 14:01:00,816 INFO  org.apache.hadoop.conf.Configuration                         [] - found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml
2023-08-30 14:01:00,949 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink.
2023-08-30 14:01:00,949 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MB may not be used by Flink.
2023-08-30 14:01:00,949 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=1728, slotsPerTaskManager=1}
2023-08-30 14:01:01,835 WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory      [] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2023-08-30 14:01:07,671 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1687922799600_0950
2023-08-30 14:01:07,749 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1687922799600_0950
2023-08-30 14:01:07,749 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
2023-08-30 14:01:07,751 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not deploy Yarn job cluster.
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
	at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:481)
	at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:81)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1956)
	at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:137)
	at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1833)
	at com.act.flink.KafkaToEsFromImageFeature.main(KafkaToEsFromImageFeature.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
	... 11 more
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
Diagnostics from YARN: Application application_1687922799600_0950 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1687922799600_0950_000001 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2023-08-30 14:01:14.268]Exception from container-launch.
Container id: container_e08_1687922799600_0950_01_000001
Exit code: 1
Exception message: Launch container failed
Shell output: main : command provided 1
main : run as user is nobody
main : requested yarn user is root
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /bigdata/hadoop/yarn/local/nmPrivate/application_1687922799600_0950/container_e08_1687922799600_0950_01_000001/container_e08_1687922799600_0950_01_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...


[2023-08-30 14:01:14.290]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

[2023-08-30 14:01:14.293]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

For more detailed output, check the application tracking page: http://nodej01:12088/cluster/app/application_1687922799600_0950 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1687922799600_0950
	at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1201)
	at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:593)
	at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:474)
	... 22 more
2023-08-30 14:01:14,668 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cancelling deployment from Deployment Failure Hook
2023-08-30 14:01:14,669 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at nodej01/192.168.71.70:8050
2023-08-30 14:01:14,690 INFO  org.apache.hadoop.yarn.client.AHSProxy                       [] - Connecting to Application History server at nodej02/192.168.71.71:10200
2023-08-30 14:01:14,691 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Killing YARN application
2023-08-30 14:01:14,718 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Killed application application_1687922799600_0950
2023-08-30 14:01:14,819 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deleting files in hdfs://nodej01:8020/user/root/.flink/application_1687922799600_0950.

2. 拉取 yarn 上的日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
yarn logs -applicationId application_1687922799600_0933

23/08/30 13:48:16 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.71.70:8050
23/08/30 13:48:17 INFO client.AHSProxy: Connecting to Application History server at node02/192.168.71.71:10200
23/08/30 13:48:18 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
23/08/30 13:48:18 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
Container: container_e08_1687922799600_0933_01_000001 on node01_45454_1693374386766
LogAggregationType: AGGREGATED
======================================================================================
LogType:container-localizer-syslog
LogLastModifiedTime:Wed Aug 30 13:46:26 +0800 2023
LogLength:349
LogContents:
2023-08-30 13:46:18,386 WARN [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Localization running as nobody not root
2023-08-30 13:46:18,969 INFO [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Disk Validator: yarn.nodemanager.disk-validator is loaded.

End of LogType:container-localizer-syslog
*******************************************************************************************

Container: container_e08_1687922799600_0933_01_000001 on node01_45454_1693374386766
LogAggregationType: AGGREGATED
======================================================================================
LogType:directory.info
LogLastModifiedTime:Wed Aug 30 13:46:26 +0800 2023
LogLength:5525
LogContents:
ls -l:
total 28
-rw-------.  1 nobody hadoop    70 Aug 30 13:46 container_tokens
lrwxrwxrwx.  1 nobody hadoop   164 Aug 30 13:46 flink-conf.yaml -> /bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/18/application_1687922799600_0933-flink-conf.yaml5969087029126036280.tmp
lrwxrwxrwx.  1 nobody hadoop   121 Aug 30 13:46 flink-dist_2.12-1.13.6.jar -> /bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/22/flink-dist_2.12-1.13.6.jar
lrwxrwxrwx.  1 nobody hadoop   135 Aug 30 13:46 flink--jar-with-dependencies.jar -> /bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/21/flink--jar-with-dependencies.jar
lrwxrwxrwx.  1 nobody hadoop   148 Aug 30 13:46 job.graph -> /bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/23/application_1687922799600_09336955381262010602857.tmp
-rwx------.  1 nobody hadoop 17023 Aug 30 13:46 launch_container.sh
drwxr-s---.  2 nobody hadoop  4096 Aug 30 13:46 lib
lrwxrwxrwx.  1 nobody hadoop   111 Aug 30 13:46 log4j.properties -> /bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/16/log4j.properties
drwxr-s---. 10 nobody hadoop   250 Aug 30 13:46 plugins
drwxr-s---.  2 nobody hadoop     6 Aug 30 13:46 tmp
find -L . -maxdepth 5 -ls:
594651    4 drwxr-s---   5 nobody   hadoop       4096 Aug 30 13:46 .
2148617758    0 drwxr-s---   2 nobody   hadoop          6 Aug 30 13:46 ./tmp
594654   20 -rwx------   1 nobody   hadoop      17023 Aug 30 13:46 ./launch_container.sh
594655    4 -rw-------   1 nobody   hadoop         70 Aug 30 13:46 ./container_tokens
19327523668   60 -r-x------   1 nobody   nobody      59672 Aug 30 13:46 ./job.graph
15042320357 104048 -r-x------   1 nobody   nobody   106542761 Aug 30 13:46 ./flink-dist_2.12-1.13.6.jar
24629443601    4 -r-x------   1 nobody   nobody       2723 Aug 30 13:46 ./log4j.properties
4300192393    0 drwxr-s---  10 nobody   hadoop        250 Aug 30 13:46 ./plugins
6444649919    0 drwxr-s---   2 nobody   hadoop         55 Aug 30 13:46 ./plugins/metrics-graphite
19332161598  172 -r-x------   1 nobody   nobody     175139 Aug 30 13:46 ./plugins/metrics-graphite/flink-metrics-graphite-1.13.6.jar
8590259050    0 drwxr-s---   2 nobody   hadoop         55 Aug 30 13:46 ./plugins/metrics-influx
6444649911  956 -r-x------   1 nobody   nobody     978710 Aug 30 13:46 ./plugins/metrics-influx/flink-metrics-influxdb-1.13.6.jar
12885353448    0 drwxr-s---   2 nobody   hadoop         57 Aug 30 13:46 ./plugins/metrics-prometheus
2148617735  100 -r-x------   1 nobody   nobody     101123 Aug 30 13:46 ./plugins/metrics-prometheus/flink-metrics-prometheus-1.13.6.jar
15042320386    0 drwxr-s---   2 nobody   hadoop         53 Aug 30 13:46 ./plugins/metrics-statsd
24629443612   12 -r-x------   1 nobody   nobody      11902 Aug 30 13:46 ./plugins/metrics-statsd/flink-metrics-statsd-1.13.6.jar
17186080326    0 drwxr-s---   2 nobody   hadoop         50 Aug 30 13:46 ./plugins/metrics-jmx
15042320383   20 -r-x------   1 nobody   nobody      17973 Aug 30 13:46 ./plugins/metrics-jmx/flink-metrics-jmx-1.13.6.jar
19332201120    0 drwxr-s---   2 nobody   hadoop        130 Aug 30 13:46 ./plugins/external-resource-gpu
2148617741    4 -r-x------   1 nobody   nobody       3189 Aug 30 13:46 ./plugins/external-resource-gpu/gpu-discovery-common.sh
24629443608    4 -r-x------   1 nobody   nobody       1794 Aug 30 13:46 ./plugins/external-resource-gpu/nvidia-gpu-discovery.sh
27920791832   16 -r-x------   1 nobody   nobody      15741 Aug 30 13:46 ./plugins/external-resource-gpu/flink-external-resource-gpu-1.13.6.jar
24629443604    0 drwxr-s---   2 nobody   hadoop         52 Aug 30 13:46 ./plugins/metrics-slf4j
19329304621   12 -r-x------   1 nobody   nobody      10032 Aug 30 13:46 ./plugins/metrics-slf4j/flink-metrics-slf4j-1.13.6.jar
24629443615    0 drwxr-s---   2 nobody   hadoop         54 Aug 30 13:46 ./plugins/metrics-datadog
6444649917  492 -r-x------   1 nobody   nobody     503276 Aug 30 13:46 ./plugins/metrics-datadog/flink-metrics-datadog-1.13.6.jar
10738166835    4 -r-x------   1 nobody   nobody        654 Aug 30 13:46 ./plugins/README.txt
10738166837    4 drwxr-s---   2 nobody   hadoop       4096 Aug 30 13:46 ./lib
27920791821 37720 -r-x------   1 nobody   nobody   38622330 Aug 30 13:46 ./lib/flink-table-blink_2.12-1.13.6.jar
15042320353  204 -r-x------   1 nobody   nobody     208006 Aug 30 13:46 ./lib/log4j-1.2-api-2.17.1.jar
32213151194  148 -r-x------   1 nobody   nobody     148127 Aug 30 13:46 ./lib/flink-json-1.13.6.jar
10738166819   24 -r-x------   1 nobody   nobody      24279 Aug 30 13:46 ./lib/log4j-slf4j-impl-2.17.1.jar
6444649914   92 -r-x------   1 nobody   nobody      92314 Aug 30 13:46 ./lib/flink-csv-1.13.6.jar
27917294447 7532 -r-x------   1 nobody   nobody    7709740 Aug 30 13:46 ./lib/flink-shaded-zookeeper-3.4.14.jar
2148617738  296 -r-x------   1 nobody   nobody     301872 Aug 30 13:46 ./lib/log4j-api-2.17.1.jar
25770817645 34236 -r-x------   1 nobody   nobody   35053606 Aug 30 13:46 ./lib/flink-table_2.12-1.13.6.jar
27920791835 1752 -r-x------   1 nobody   nobody    1790452 Aug 30 13:46 ./lib/log4j-core-2.17.1.jar
32213151191    4 -r-x------   1 nobody   nobody       1042 Aug 30 13:46 ./flink-conf.yaml
10738166824 120580 -r-x------   1 nobody   nobody   123473504 Aug 30 13:46 ./flink--jar-with-dependencies.jar
broken symlinks(find -L . -maxdepth 5 -type l -ls):

End of LogType:directory.info
*******************************************************************************

Container: container_e08_1687922799600_0933_01_000001 on node01_45454_1693374386766
LogAggregationType: AGGREGATED
======================================================================================
LogType:jobmanager.err
LogLastModifiedTime:Wed Aug 30 13:46:26 +0800 2023
LogLength:1163
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/13/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.logging.log4j.spi.LoggerContextFactory.isClassLoaderDependent()Z
	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:49)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)

End of LogType:jobmanager.err
*******************************************************************************


End of LogType:jobmanager.out
*******************************************************************************

Container: container_e08_1687922799600_0933_01_000001 on node01_45454_1693374386766
LogAggregationType: AGGREGATED
======================================================================================
LogType:launch_container.sh
LogLastModifiedTime:Wed Aug 30 13:46:26 +0800 2023
LogLength:17023
LogContents:
#!/bin/bash

set -o pipefail -e
export PRELAUNCH_OUT="/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/prelaunch.out"
exec >"${PRELAUNCH_OUT}"
export PRELAUNCH_ERR="/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/prelaunch.err"
exec 2>"${PRELAUNCH_ERR}"
echo "Setting up env variables"
export JAVA_HOME=${JAVA_HOME:-"/usr/local/jdk1.8.0_191"}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/3.1.0.0-78/hadoop/conf"}
export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/3.1.0.0-78/hadoop-yarn"}
export HADOOP_HOME=${HADOOP_HOME:-"/usr/hdp/3.1.0.0-78/hadoop"}
export PATH=${PATH:-"/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/jdk1.8.0_191/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent"}
export LANG=${LANG:-"en_US.UTF-8"}
export HADOOP_TOKEN_FILE_LOCATION="/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/container_tokens"
export CONTAINER_ID="container_e08_1687922799600_0933_01_000001"
export NM_PORT="45454"
export NM_HOST="node01"
export NM_HTTP_PORT="8042"
export LOCAL_DIRS="/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933"
export LOCAL_USER_DIRS="/bigdata/hadoop/yarn/local/usercache/root/"
export LOG_DIRS="/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001"
export USER="root"
export LOGNAME="root"
export HOME="/home/"
export PWD="/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001"
export JVM_PID="$$"
export MALLOC_ARENA_MAX="4"
export NM_AUX_SERVICE_timeline_collector=""
export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
export NM_AUX_SERVICE_spark2_shuffle=""
export _CLIENT_HOME_DIR="hdfs://node01:8020/user/root"
export APP_SUBMIT_TIME_ENV="1693374374141"
export _APP_ID="application_1687922799600_0933"
export HADOOP_USER_NAME="root"
export TIMELINE_FLOW_NAME_TAG="flink-dwd_picture_features_to_es"
export TIMELINE_FLOW_VERSION_TAG="1"
export APPLICATION_WEB_PROXY_BASE="/proxy/application_1687922799600_0933"
export _CLIENT_SHIP_FILES="YarnLocalResourceDescriptor{key=log4j.properties, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/log4j.properties, size=2723, modificationTime=1693374368642, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/log4j-1.2-api-2.17.1.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/log4j-1.2-api-2.17.1.jar, size=208006, modificationTime=1693374368718, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/log4j-api-2.17.1.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/log4j-api-2.17.1.jar, size=301872, modificationTime=1693374368750, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/log4j-core-2.17.1.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/log4j-core-2.17.1.jar, size=1790452, modificationTime=1693374368813, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/log4j-slf4j-impl-2.17.1.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/log4j-slf4j-impl-2.17.1.jar, size=24279, modificationTime=1693374368849, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/flink-json-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/flink-json-1.13.6.jar, size=148127, modificationTime=1693374368877, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/flink-shaded-zookeeper-3.4.14.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/flink-shaded-zookeeper-3.4.14.jar, size=7709740, modificationTime=1693374369017, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/flink-table_2.12-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/flink-table_2.12-1.13.6.jar, size=35053606, modificationTime=1693374369550, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/flink-csv-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/flink-csv-1.13.6.jar, size=92314, modificationTime=1693374369579, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=lib/flink-table-blink_2.12-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/lib/flink-table-blink_2.12-1.13.6.jar, size=38622330, modificationTime=1693374370138, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/external-resource-gpu/flink-external-resource-gpu-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/external-resource-gpu/flink-external-resource-gpu-1.13.6.jar, size=15741, modificationTime=1693374370208, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/external-resource-gpu/gpu-discovery-common.sh, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/external-resource-gpu/gpu-discovery-common.sh, size=3189, modificationTime=1693374370243, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/external-resource-gpu/nvidia-gpu-discovery.sh, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/external-resource-gpu/nvidia-gpu-discovery.sh, size=1794, modificationTime=1693374370267, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-datadog/flink-metrics-datadog-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-datadog/flink-metrics-datadog-1.13.6.jar, size=503276, modificationTime=1693374370298, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-graphite/flink-metrics-graphite-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-graphite/flink-metrics-graphite-1.13.6.jar, size=175139, modificationTime=1693374370333, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-influx/flink-metrics-influxdb-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-influx/flink-metrics-influxdb-1.13.6.jar, size=978710, modificationTime=1693374370445, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-jmx/flink-metrics-jmx-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-jmx/flink-metrics-jmx-1.13.6.jar, size=17973, modificationTime=1693374370470, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-prometheus/flink-metrics-prometheus-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-prometheus/flink-metrics-prometheus-1.13.6.jar, size=101123, modificationTime=1693374370490, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-slf4j/flink-metrics-slf4j-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-slf4j/flink-metrics-slf4j-1.13.6.jar, size=10032, modificationTime=1693374370516, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/metrics-statsd/flink-metrics-statsd-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/metrics-statsd/flink-metrics-statsd-1.13.6.jar, size=11902, modificationTime=1693374370565, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=plugins/README.txt, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/plugins/README.txt, size=654, modificationTime=1693374370587, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=flink--jar-with-dependencies.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/flink--jar-with-dependencies.jar, size=123473504, modificationTime=1693374372495, visibility=APPLICATION, type=FILE};YarnLocalResourceDescriptor{key=flink-conf.yaml, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/application_1687922799600_0933-flink-conf.yaml5969087029126036280.tmp, size=1042, modificationTime=1693374374089, visibility=APPLICATION, type=FILE}"
export CLASSPATH=":flink--jar-with-dependencies.jar:lib/flink-csv-1.13.6.jar:lib/flink-json-1.13.6.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.13.6.jar:lib/flink-table_2.12-1.13.6.jar:lib/log4j-1.2-api-2.17.1.jar:lib/log4j-api-2.17.1.jar:lib/log4j-core-2.17.1.jar:lib/log4j-slf4j-impl-2.17.1.jar:flink-dist_2.12-1.13.6.jar:job.graph:flink-conf.yaml::$HADOOP_CONF_DIR:/usr/hdp/3.1.0.0-78/hadoop/*:/usr/hdp/3.1.0.0-78/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*"
export _FLINK_YARN_FILES="hdfs://node01:8020/user/root/.flink/application_1687922799600_0933"
export _FLINK_CLASSPATH=":flink--jar-with-dependencies.jar:lib/flink-csv-1.13.6.jar:lib/flink-json-1.13.6.jar:lib/flink-shaded-zookeeper-3.4.14.jar:lib/flink-table-blink_2.12-1.13.6.jar:lib/flink-table_2.12-1.13.6.jar:lib/log4j-1.2-api-2.17.1.jar:lib/log4j-api-2.17.1.jar:lib/log4j-core-2.17.1.jar:lib/log4j-slf4j-impl-2.17.1.jar:flink-dist_2.12-1.13.6.jar:job.graph:flink-conf.yaml:"
export TIMELINE_FLOW_RUN_ID_TAG="1693374374196"
export _FLINK_DIST_JAR="YarnLocalResourceDescriptor{key=flink-dist_2.12-1.13.6.jar, path=hdfs://node01:8020/user/root/.flink/application_1687922799600_0933/flink-dist_2.12-1.13.6.jar, size=106542761, modificationTime=1693374374000, visibility=APPLICATION, type=FILE}"
echo "Setting up job resources"
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/23/application_1687922799600_09336955381262010602857.tmp" "job.graph"
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/22/flink-dist_2.12-1.13.6.jar" "flink-dist_2.12-1.13.6.jar"
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/16/log4j.properties" "log4j.properties"
mkdir -p plugins/metrics-graphite
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/15/flink-metrics-graphite-1.13.6.jar" "plugins/metrics-graphite/flink-metrics-graphite-1.13.6.jar"
mkdir -p plugins/metrics-influx
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/12/flink-metrics-influxdb-1.13.6.jar" "plugins/metrics-influx/flink-metrics-influxdb-1.13.6.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/10/flink-table-blink_2.12-1.13.6.jar" "lib/flink-table-blink_2.12-1.13.6.jar"
mkdir -p plugins/metrics-prometheus
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/11/flink-metrics-prometheus-1.13.6.jar" "plugins/metrics-prometheus/flink-metrics-prometheus-1.13.6.jar"
mkdir -p plugins/metrics-statsd
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/32/flink-metrics-statsd-1.13.6.jar" "plugins/metrics-statsd/flink-metrics-statsd-1.13.6.jar"
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/18/application_1687922799600_0933-flink-conf.yaml5969087029126036280.tmp" "flink-conf.yaml"
mkdir -p plugins/metrics-jmx
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/30/flink-metrics-jmx-1.13.6.jar" "plugins/metrics-jmx/flink-metrics-jmx-1.13.6.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/14/log4j-1.2-api-2.17.1.jar" "lib/log4j-1.2-api-2.17.1.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/26/flink-json-1.13.6.jar" "lib/flink-json-1.13.6.jar"
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/21/flink--jar-with-dependencies.jar" "flink--jar-with-dependencies.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/13/log4j-slf4j-impl-2.17.1.jar" "lib/log4j-slf4j-impl-2.17.1.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/20/flink-csv-1.13.6.jar" "lib/flink-csv-1.13.6.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/34/flink-shaded-zookeeper-3.4.14.jar" "lib/flink-shaded-zookeeper-3.4.14.jar"
mkdir -p plugins/external-resource-gpu
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/27/gpu-discovery-common.sh" "plugins/external-resource-gpu/gpu-discovery-common.sh"
mkdir -p plugins/metrics-slf4j
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/31/flink-metrics-slf4j-1.13.6.jar" "plugins/metrics-slf4j/flink-metrics-slf4j-1.13.6.jar"
mkdir -p plugins/external-resource-gpu
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/24/nvidia-gpu-discovery.sh" "plugins/external-resource-gpu/nvidia-gpu-discovery.sh"
mkdir -p plugins/metrics-datadog
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/28/flink-metrics-datadog-1.13.6.jar" "plugins/metrics-datadog/flink-metrics-datadog-1.13.6.jar"
mkdir -p plugins
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/29/README.txt" "plugins/README.txt"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/19/log4j-api-2.17.1.jar" "lib/log4j-api-2.17.1.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/33/flink-table_2.12-1.13.6.jar" "lib/flink-table_2.12-1.13.6.jar"
mkdir -p plugins/external-resource-gpu
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/17/flink-external-resource-gpu-1.13.6.jar" "plugins/external-resource-gpu/flink-external-resource-gpu-1.13.6.jar"
mkdir -p lib
ln -sf "/bigdata/hadoop/yarn/local/usercache/root/appcache/application_1687922799600_0933/filecache/25/log4j-core-2.17.1.jar" "lib/log4j-core-2.17.1.jar"
echo "Copying debugging information"
# Creating copy of launch script
cp "launch_container.sh" "/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/launch_container.sh"
chmod 640 "/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/launch_container.sh"
# Determining directory contents
echo "ls -l:" 1>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
ls -l 1>>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
find -L . -maxdepth 5 -ls 1>>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/directory.info"
echo "Launching container"
exec /bin/bash -c "$JAVA_HOME/bin/java -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file="/bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/jobmanager.log" -Dlog4j.configuration=file:log4j.properties -Dlog4j.configurationFile=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b 1> /bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/jobmanager.out 2> /bigdata/hadoop/yarn/log/application_1687922799600_0933/container_e08_1687922799600_0933_01_000001/jobmanager.err"

End of LogType:launch_container.sh
************************************************************************************


End of LogType:prelaunch.err
******************************************************************************

Container: container_e08_1687922799600_0933_01_000001 on node01_45454_1693374386766
LogAggregationType: AGGREGATED
======================================================================================
LogType:prelaunch.out
LogLastModifiedTime:Wed Aug 30 13:46:26 +0800 2023
LogLength:100
LogContents:
Setting up env variables
Setting up job resources
Copying debugging information
Launching container

End of LogType:prelaunch.out
******************************************************************************

分析出是 jar包异常后， 打印依赖树

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
mvn dependency:tree  -Dverbose

[INFO] ---------------------< com.hnbian:flink >---------------------
[INFO] Building flink                                             [4/6]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.2.0:tree (default-cli) @ flink ---
[INFO] com.hnbian.:flink:jar:
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- org.apache.flink:flink-connector-elasticsearch7_2.12:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-connector-elasticsearch-base_2.12:jar:1.13.6:compile
[INFO] |  +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:7.5.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch:jar:7.5.1:compile
[INFO] |  |  |  +- org.elasticsearch:elasticsearch-core:jar:7.5.1:compile
[INFO] |  |  |  +- org.elasticsearch:elasticsearch-secure-sm:jar:7.5.1:compile
[INFO] |  |  |  +- org.elasticsearch:elasticsearch-x-content:jar:7.5.1:compile
[INFO] |  |  |  |  +- org.yaml:snakeyaml:jar:1.17:compile
[INFO] |  |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.8.11:compile
[INFO] |  |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.8.11:compile
[INFO] |  |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.11:compile
[INFO] |  |  |  +- org.elasticsearch:elasticsearch-geo:jar:7.5.1:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-core:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-analyzers-common:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-backward-codecs:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-grouping:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-highlighter:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-join:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-memory:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-misc:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-queries:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-queryparser:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-sandbox:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-spatial:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-spatial-extras:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-spatial3d:jar:8.3.0:compile
[INFO] |  |  |  +- org.apache.lucene:lucene-suggest:jar:8.3.0:compile
[INFO] |  |  |  +- org.elasticsearch:elasticsearch-cli:jar:7.5.1:compile
[INFO] |  |  |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile
[INFO] |  |  |  +- com.carrotsearch:hppc:jar:0.8.1:compile
[INFO] |  |  |  +- com.tdunning:t-digest:jar:3.2:compile
[INFO] |  |  |  +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile #log4j
[INFO] |  |  |  \- org.elasticsearch:jna:jar:4.5.1:compile
[INFO] |  |  +- org.elasticsearch.client:elasticsearch-rest-client:jar:7.5.1:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile
[INFO] |  |  |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.12:compile
[INFO] |  |  +- org.elasticsearch.plugin:mapper-extras-client:jar:7.5.1:compile
[INFO] |  |  +- org.elasticsearch.plugin:parent-join-client:jar:7.5.1:compile
[INFO] |  |  +- org.elasticsearch.plugin:aggs-matrix-stats-client:jar:7.5.1:compile
[INFO] |  |  +- org.elasticsearch.plugin:rank-eval-client:jar:7.5.1:compile
[INFO] |  |  \- org.elasticsearch.plugin:lang-mustache-client:jar:7.5.1:compile
[INFO] |  |     \- com.github.spullara.mustache.java:compiler:jar:0.9.6:compile
[INFO] |  \- org.apache.flink:force-shading:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-core:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-annotations:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-shaded-asm-7:jar:7.1-13.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  +- org.apache.flink:flink-shaded-guava:jar:18.0-13.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.15:compile
[INFO] |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] +- org.apache.flink:flink-streaming-java_2.12:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-file-sink-common:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-runtime_2.12:jar:1.13.6:compile
[INFO] |  |  +- org.apache.flink:flink-queryable-state-client-java:jar:1.13.6:compile
[INFO] |  |  +- org.apache.flink:flink-hadoop-fs:jar:1.13.6:compile
[INFO] |  |  +- org.apache.flink:flink-shaded-netty:jar:4.1.49.Final-13.0:compile
[INFO] |  |  +- org.apache.flink:flink-shaded-jackson:jar:2.12.1-13.0:compile
[INFO] |  |  +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.4.14-13.0:compile
[INFO] |  |  +- org.javassist:javassist:jar:3.24.0-GA:compile
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.12.7:compile
[INFO] |  |  +- com.typesafe.akka:akka-actor_2.12:jar:2.5.21:compile
[INFO] |  |  |  +- com.typesafe:config:jar:1.3.3:compile
[INFO] |  |  |  \- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.8.0:compile
[INFO] |  |  +- com.typesafe.akka:akka-stream_2.12:jar:2.5.21:compile
[INFO] |  |  |  +- org.reactivestreams:reactive-streams:jar:1.0.2:compile
[INFO] |  |  |  \- com.typesafe:ssl-config-core_2.12:jar:0.3.7:compile
[INFO] |  |  |     \- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.1:compile
[INFO] |  |  +- com.typesafe.akka:akka-protobuf_2.12:jar:2.5.21:compile
[INFO] |  |  +- com.typesafe.akka:akka-slf4j_2.12:jar:2.5.21:compile
[INFO] |  |  +- org.clapper:grizzled-slf4j_2.12:jar:1.3.2:compile
[INFO] |  |  +- com.github.scopt:scopt_2.12:jar:3.5.0:compile
[INFO] |  |  +- com.twitter:chill_2.12:jar:0.7.6:compile
[INFO] |  |  |  \- com.twitter:chill-java:jar:0.7.6:compile
[INFO] |  |  \- org.lz4:lz4-java:jar:1.6.0:compile
[INFO] |  +- org.apache.flink:flink-java:jar:1.13.6:compile
[INFO] |  \- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] +- org.apache.flink:flink-connector-kafka_2.12:jar:1.13.6:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:2.4.1:compile
[INFO] |  |  \- com.github.luben:zstd-jni:jar:1.4.3-1:compile
[INFO] |  \- org.apache.flink:flink-connector-base:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-json:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-avro:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-connector-jdbc_2.12:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-table-api-java-bridge_2.12:jar:1.13.6:compile
[INFO] |  \- org.apache.flink:flink-table-api-java:jar:1.13.6:compile
[INFO] |     \- org.apache.flink:flink-table-common:jar:1.13.6:compile
[INFO] |        \- org.apache.flink:flink-connector-files:jar:1.13.6:compile
[INFO] +- org.apache.flink:flink-parquet_2.12:jar:1.13.6:compile
[INFO] +- mysql:mysql-connector-java:jar:8.0.26:compile
[INFO] |  \- com.google.protobuf:protobuf-java:jar:3.11.4:compile
[INFO] +- org.apache.flink:flink-clients_2.12:jar:1.13.6:compile
[INFO] |  +- org.apache.flink:flink-optimizer_2.12:jar:1.13.6:compile
[INFO] |  \- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] +- joda-time:joda-time:jar:2.9.6:compile
[INFO] +- redis.clients:jedis:jar:2.9.0:compile
[INFO] |  \- org.apache.commons:commons-pool2:jar:2.4.2:compile
[INFO] +- org.apache.parquet:parquet-common:jar:1.11.0:compile
[INFO] |  +- org.apache.parquet:parquet-format-structures:jar:1.11.0:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  \- org.apache.yetus:audience-annotations:jar:0.11.0:compile
[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  \- org.tukaani:xz:jar:1.5:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.11.0:compile
[INFO] |  \- org.apache.parquet:parquet-column:jar:1.11.0:compile
[INFO] |     \- org.apache.parquet:parquet-encoding:jar:1.11.0:compile
[INFO] +- org.apache.parquet:parquet-hadoop:jar:1.11.1:compile
[INFO] |  +- org.apache.parquet:parquet-jackson:jar:1.11.1:compile
[INFO] |  \- commons-pool:commons-pool:jar:1.6:compile
[INFO] +- com.alibaba:fastjson:jar:2.0.6:compile
[INFO] |  \- com.alibaba.fastjson2:fastjson2-extension:jar:2.0.6:compile
[INFO] |     \- com.alibaba.fastjson2:fastjson2:jar:2.0.6:compile
[INFO] +- commons-lang:commons-lang:jar:2.6:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.1.0:provided
[INFO] |  +- com.google.guava:guava:jar:11.0.2:provided
[INFO] |  +- commons-io:commons-io:jar:2.5:compile
[INFO] |  +- commons-net:commons-net:jar:3.6:provided
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- org.eclipse.jetty:jetty-server:jar:9.3.19.v20170502:provided
[INFO] |  |  +- org.eclipse.jetty:jetty-http:jar:9.3.19.v20170502:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-io:jar:9.3.19.v20170502:provided
[INFO] |  +- org.eclipse.jetty:jetty-util:jar:9.3.19.v20170502:provided
[INFO] |  +- org.eclipse.jetty:jetty-servlet:jar:9.3.19.v20170502:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-security:jar:9.3.19.v20170502:provided
[INFO] |  +- org.eclipse.jetty:jetty-webapp:jar:9.3.19.v20170502:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-xml:jar:9.3.19.v20170502:provided
[INFO] |  +- javax.servlet.jsp:jsp-api:jar:2.1:provided
[INFO] |  +- com.sun.jersey:jersey-core:jar:1.19:provided
[INFO] |  |  \- javax.ws.rs:jsr311-api:jar:1.1.1:provided
[INFO] |  +- com.sun.jersey:jersey-servlet:jar:1.19:provided
[INFO] |  +- com.sun.jersey:jersey-json:jar:1.19:provided
[INFO] |  |  +- org.codehaus.jettison:jettison:jar:1.1:provided
[INFO] |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
[INFO] |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.2:provided
[INFO] |  |  \- org.codehaus.jackson:jackson-xc:jar:1.9.2:provided
[INFO] |  +- com.sun.jersey:jersey-server:jar:1.19:provided
[INFO] |  +- log4j:log4j:jar:1.2.17:provided # log4j 
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.3:provided
[INFO] |  +- org.apache.commons:commons-configuration2:jar:2.1.1:provided
[INFO] |  +- com.google.re2j:re2j:jar:1.1:provided
[INFO] |  +- com.google.code.gson:gson:jar:2.2.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:3.1.0:provided
[INFO] |  |  +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
[INFO] |  |  |  \- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
[INFO] |  |  +- net.minidev:json-smart:jar:2.3:provided
[INFO] |  |  |  \- net.minidev:accessors-smart:jar:1.2:provided
[INFO] |  |  |     \- org.ow2.asm:asm:jar:5.0.4:provided
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.12.0:provided
[INFO] |  +- com.jcraft:jsch:jar:0.1.54:provided
[INFO] |  +- org.apache.curator:curator-client:jar:2.12.0:provided
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.12.0:provided
[INFO] |  +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.9:provided
[INFO] |  |  \- jline:jline:jar:0.9.94:provided
[INFO] |  +- org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
[INFO] |  |  +- org.apache.kerby:kerb-client:jar:1.0.1:provided
[INFO] |  |  |  +- org.apache.kerby:kerby-config:jar:1.0.1:provided
[INFO] |  |  |  +- org.apache.kerby:kerb-core:jar:1.0.1:provided
[INFO] |  |  |  |  \- org.apache.kerby:kerby-pkix:jar:1.0.1:provided
[INFO] |  |  |  |     +- org.apache.kerby:kerby-asn1:jar:1.0.1:provided
[INFO] |  |  |  |     \- org.apache.kerby:kerby-util:jar:1.0.1:provided
[INFO] |  |  |  +- org.apache.kerby:kerb-common:jar:1.0.1:provided
[INFO] |  |  |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.1:provided
[INFO] |  |  |  +- org.apache.kerby:kerb-util:jar:1.0.1:provided
[INFO] |  |  |  \- org.apache.kerby:token-provider:jar:1.0.1:provided
[INFO] |  |  \- org.apache.kerby:kerb-admin:jar:1.0.1:provided
[INFO] |  |     +- org.apache.kerby:kerb-server:jar:1.0.1:provided
[INFO] |  |     |  \- org.apache.kerby:kerb-identity:jar:1.0.1:provided
[INFO] |  |     \- org.apache.kerby:kerby-xdr:jar:1.0.1:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.8:provided
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.7.8:compile
[INFO] |  +- org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
[INFO] |  \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
[INFO] +- org.apache.hadoop:hadoop-annotations:jar:3.1.0:compile
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:3.1.0:provided
[INFO] |  +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:provided
[INFO] |  +- commons-daemon:commons-daemon:jar:1.0.13:provided
[INFO] |  +- io.netty:netty:jar:3.10.5.Final:provided
[INFO] |  +- io.netty:netty-all:jar:4.0.52.Final:provided
[INFO] |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] \- org.apache.hadoop:hadoop-client:jar:3.1.0:provided
[INFO]    +- org.apache.hadoop:hadoop-hdfs-client:jar:3.1.0:provided
[INFO]    |  \- com.squareup.okhttp:okhttp:jar:2.7.5:provided
[INFO]    |     \- com.squareup.okio:okio:jar:1.6.0:provided
[INFO]    +- org.apache.hadoop:hadoop-yarn-api:jar:3.1.0:provided
[INFO]    |  \- javax.xml.bind:jaxb-api:jar:2.2.11:provided
[INFO]    +- org.apache.hadoop:hadoop-yarn-client:jar:3.1.0:provided
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.1.0:provided
[INFO]    |  \- org.apache.hadoop:hadoop-yarn-common:jar:3.1.0:provided
[INFO]    |     +- com.sun.jersey:jersey-client:jar:1.19:provided
[INFO]    |     +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.7.8:provided
[INFO]    |     \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
[INFO]    |        \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
[INFO]    \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.1.0:provided
[INFO]       \- org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.1.0:provided
[INFO]

3. 找到冲突包，并排除

1
2
3
4
5
6
7
8
9
10
11
12
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch7_2.12</artifactId>
<version>${flink.version}</version>
    <exclusions>
        <!-- Exclude log4j-api -->
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
        </exclusion>
    </exclusions>
</dependency>

4. 重新打包部署， 解决问题

完美执行

文章作者: hnbian
文章链接: https://www.hnbian.cn/posts/be4ab507.html
版权声明: 本博客所有文章除特別声明外，均采用 CC BY 4.0 许可协议。转载请注明来源 hnbian !
exception flink
评论
  目录