Ambari 添加 flink 服务

1. 安装 git

# 检查是否安装 git
[root@node1 ~]# git --help
-bash: git: 未找到命令

# 如果没有安装 git 则需要先安装git
[root@node1 ~]# yum install git
已加载插件：fastestmirror
Repository base is listed more than once in the configuration
Determining fastest mirrors
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
......
软件包 git-1.8.3.1-23.el7_8.x86_64 已安装并且是最新版本
无须任何处理
您在 /var/spool/mail/root 中有新邮件

2. 设置 version 变量

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'` 
[root@node1 opt]# echo $VERSION
3.1

3.下载ambari-flink-service服务

# 下载ambari-flink-service服务到 ambari-server 资源目录下
[root@node1 opt]# sudo git clone https://github.com/abajwa-hw/ambari-flink-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK
正克隆到 '/var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK'...
remote: Enumerating objects: 192, done.
remote: Total 192 (delta 0), reused 0 (delta 0), pack-reused 192
接收对象中: 100% (192/192), 2.08 MiB | 35.00 KiB/s, done.
处理 delta 中: 100% (89/89), done.

[root@node1 opt]# cd /var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK/
[root@node1 FLINK]# ll
总用量 20
drwxr-xr-x 2 root root   58 2月  23 16:16 configuration
-rw-r--r-- 1 root root  223 2月  23 16:16 kerberos.json
-rw-r--r-- 1 root root 1777 2月  23 16:16 metainfo.xml
drwxr-xr-x 3 root root   21 2月  23 16:16 package
-rwxr-xr-x 1 root root 8114 2月  23 16:16 README.md
-rw-r--r-- 1 root root  125 2月  23 16:16 role_command_order.json
drwxr-xr-x 2 root root  236 2月  23 16:16 screenshots

4. 修改配置文件

1. 编辑 metainfo.xml 将安装的版本修改为 1.10.3

[root@node1 FLINK]# vim metainfo.xml

<name>FLINK</name>
<displayName>Flink</displayName>
<comment>Apache Flink is a streaming ...</comment>
<version>1.10.3</version>

2. 配置 Flink on yarn 故障转移方式

<property>
	<name>yarn.client.failover-proxy-provider</name>
	<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>

<property>
	<name>yarn.client.failover-proxy-provider</name>
	<value>org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider</value>
</property>

3. 编辑configuration/flink-ambari-config.xml修改下载地址

[root@node1 FLINK]# vim configuration/flink-ambari-config.xml

<property>
    <name>flink_download_url</name>
    <!--<value>http://www.us.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz</value>-->
    <value>https://downloads.apache.org/flink/flink-1.10.3/flink-1.10.3-bin-scala_2.11.tgz</value>
    <description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
  </property>

5.创建 Flink 用户组与用户

# 添加用户组
[root@node1 FLINK]# groupadd flink
# 添加用户
[root@node1 FLINK]# useradd  -d /home/flink  -g flink flink

6. 重启 ambari-server 并检查可用服务列表

[root@node1 ~]# ambari-server restart
Using python  /usr/bin/python
Restarting ambari-server
Waiting for server stop...
Ambari Server stopped
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.............................................
Server started listening on 8080

• 查看安装服务中是否有 Flink

7. 安装 flink

8. 启动 flink 服务

9. 测试 flink

• 提交任务

[hdfs@node1 flink-1.10.2]$ bin/flink run ./examples/batch/WordCount.jar # 提交 flink 自带的测试任务
Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/flink-1.10.2/lib/slf4j-log4j12-
...
# 执行结果
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
...

• 查看 Flink webUI 中的任务情况

10.遇到的问题

1. 启动时 java: 未找到命令

resource_management.core.exceptions.ExecutionFailed: Execution of 'export ...9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.0.0-78/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.0.0-78/tez/lib/tez.tar.gz; /usr/hdp/3.1.0.0-78/flink/bin/yarn-session.sh -n 1 -s 1 -jm 768 -tm
-qu default -nm flinkapp-from-ambari -d >> /var/log/flink/flink-setup.log' returned 127. /usr/hdp/3.1.0.0-78/flink/bin/yarn-session.sh:行37: java: 未找到命令

解决办法：

2. IllegalConfigurationException:

org.apache.flink.configuration.IllegalConfigurationException: Sum of configured Framework Heap Memory (128.000mb (134217728 bytes)), Framework Off-Heap Memory (128.000mb (134217728 bytes)), Task Off-Heap Memory (0 bytes), Managed Memory (25.600mb (26843546 bytes)) and Network Memory (64.000mb (67108864 bytes)) exceed configured Total Flink Memory (64.000mb (67108864 bytes)).
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveInternalMemoryFromTotalFlinkMemory(TaskExecutorProcessUtils.java:320)
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveProcessSpecWithTotalProcessMemory(TaskExecutorProcessUtils.java:248)
	at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.processSpecFromConfig(TaskExecutorProcessUtils.java:147)
	at org.apache.flink.client.deployment.AbstractContainerizedClusterClientFactory.getClusterSpecification(AbstractContainerizedClusterClientFactory.java:44)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:547)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$5(FlinkYarnSessionCli.java:786)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:786)

解决办法：

3. 页面显示异常

从页面启动 flink之后，Flink-yarn-session 启动成功，但是 ambari 进度不到 100%

4. 页面没有 Flink web ui 链接地址