Ambari 添加 flink 服务


1. 安装 git

# 检查是否安装 git
[root@node1 ~]# git --help
-bash: git: 未找到命令

# 如果没有安装 git 则需要先安装git
[root@node1 ~]# yum install git
已加载插件:fastestmirror
Repository base is listed more than once in the configuration
Determining fastest mirrors
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
......
软件包 git-1.8.3.1-23.el7_8.x86_64 已安装并且是最新版本
无须任何处理
您在 /var/spool/mail/root 中有新邮件

2. 设置 version 变量

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'` 
[root@node1 opt]# echo $VERSION
3.1

# 下载ambari-flink-service服务到 ambari-server 资源目录下
[root@node1 opt]# sudo git clone https://github.com/abajwa-hw/ambari-flink-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK
正克隆到 '/var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK'...
remote: Enumerating objects: 192, done.
remote: Total 192 (delta 0), reused 0 (delta 0), pack-reused 192
接收对象中: 100% (192/192), 2.08 MiB | 35.00 KiB/s, done.
处理 delta 中: 100% (89/89), done.

[root@node1 opt]# cd /var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK/
[root@node1 FLINK]# ll
总用量 20
drwxr-xr-x 2 root root   58 2月  23 16:16 configuration
-rw-r--r-- 1 root root  223 2月  23 16:16 kerberos.json
-rw-r--r-- 1 root root 1777 2月  23 16:16 metainfo.xml
drwxr-xr-x 3 root root   21 2月  23 16:16 package
-rwxr-xr-x 1 root root 8114 2月  23 16:16 README.md
-rw-r--r-- 1 root root  125 2月  23 16:16 role_command_order.json
drwxr-xr-x 2 root root  236 2月  23 16:16 screenshots

4. 修改配置文件

  1. 编辑 metainfo.xml 将安装的版本修改为 1.10.3

[root@node1 FLINK]# vim metainfo.xml

<name>FLINK</name>
<displayName>Flink</displayName>
<comment>Apache Flink is a streaming ...</comment>
<version>1.10.3</version>
  1. 配置 Flink on yarn 故障转移方式
<property>
    <name>yarn.client.failover-proxy-provider</name>
    <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>
<property>
    <name>yarn.client.failover-proxy-provider</name>
    <value>org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider</value>
</property>
  1. 编辑configuration/flink-ambari-config.xml修改下载地址

[root@node1 FLINK]# vim configuration/flink-ambari-config.xml

<property>
    <name>flink_download_url</name>
    <!--<value>http://www.us.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz</value>-->
    <value>https://downloads.apache.org/flink/flink-1.10.3/flink-1.10.3-bin-scala_2.11.tgz</value>
    <description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
  </property>
# 添加用户组
[root@node1 FLINK]# groupadd flink
# 添加用户
[root@node1 FLINK]# useradd  -d /home/flink  -g flink flink

6. 重启 ambari-server 并检查可用服务列表

[root@node1 ~]# ambari-server restart
Using python  /usr/bin/python
Restarting ambari-server
Waiting for server stop...
Ambari Server stopped
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.............................................
Server started listening on 8080
  • 查看安装服务中是否有 Flink

Flink 已经添加到可用服务列表

1.选择 Flink 添加服务

2.确认依赖服务是否安装

3.选择安装服务到哪台服务器

4.修改 flink安装路径

5.安装与测试

6.安装完成

7.摘要

8.查看 ambari 中安装好的flink

9.点击上图的配置按钮,修改 Javahome 路径

9.增加堆内存

启动 flink 服务

yarn 中运行的 Flink yarn session

Flink web UI

  • 提交任务
[hdfs@node1 flink-1.10.2]$ bin/flink run ./examples/batch/WordCount.jar # 提交 flink 自带的测试任务
Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/flink-1.10.2/lib/slf4j-log4j12-
...
# 执行结果
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
...
  • 查看 Flink webUI 中的任务情况

提交任务之后马上进Flink webUI 能看到 Wordcount 任务正在运行

查看任务运行流程

10.遇到的问题

  1. 启动时 java: 未找到命令
resource_management.core.exceptions.ExecutionFailed: Execution of 'export ...9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.0.0-78/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.0.0-78/tez/lib/tez.tar.gz; /usr/hdp/3.1.0.0-78/flink/bin/yarn-session.sh -n 1 -s 1 -jm 768 -tm
-qu default -nm flinkapp-from-ambari -d >> /var/log/flink/flink-setup.log' returned 127. /usr/hdp/3.1.0.0-78/flink/bin/yarn-session.sh:37: java: 未找到命令

解决办法:

修改 Javahome 路径

  1. IllegalConfigurationException:
org.apache.flink.configuration.IllegalConfigurationException: Sum of configured Framework Heap Memory (128.000mb (134217728 bytes)), Framework Off-Heap Memory (128.000mb (134217728 bytes)), Task Off-Heap Memory (0 bytes), Managed Memory (25.600mb (26843546 bytes)) and Network Memory (64.000mb (67108864 bytes)) exceed configured Total Flink Memory (64.000mb (67108864 bytes)).
    at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveInternalMemoryFromTotalFlinkMemory(TaskExecutorProcessUtils.java:320)
    at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.deriveProcessSpecWithTotalProcessMemory(TaskExecutorProcessUtils.java:248)
    at org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils.processSpecFromConfig(TaskExecutorProcessUtils.java:147)
    at org.apache.flink.client.deployment.AbstractContainerizedClusterClientFactory.getClusterSpecification(AbstractContainerizedClusterClientFactory.java:44)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:547)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$5(FlinkYarnSessionCli.java:786)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:786)

解决办法:

增加堆内存

  1. 页面显示异常

从页面启动 flink之后,Flink-yarn-session 启动成功,但是 ambari 进度不到 100%

  1. 页面没有 Flink web ui 链接地址

文章作者: hnbian
版权声明: 本博客所有文章除特別声明外,均采用 CC BY 4.0 许可协议。转载请注明来源 hnbian !
评论
 上一篇
TPC-H 测试工具介绍 TPC-H 测试工具介绍
1. TPC简介TPC 即事务处理性能委员会( Transaction ProcessingPerformance Council ,官网:http://tpc.org),是由数10家会员公司创建的非盈利组织,总部设在美国。TPC的成员主要
2021-03-04
下一篇 
Azkaban 2. 服务部署 Azkaban 2. 服务部署
1. 编译源码 下载最新版代码 wget https://codeload.github.com/azkaban/azkaban/tar.gz/3.85.1 [root@hnode1 opt]# wget https://codelo
2021-02-10
  目录