Hadoop生态圈-Azkaban部署实战
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.Azkaban部署流程
1>.上传azkaban程序并创建解压目录
[yinzhengjie@s101 data]$ pwd/home/yinzhengjie/data[yinzhengjie@s101 data]$ lltotal 401716-rw-r--r-- 1 yinzhengjie yinzhengjie 16664491 Apr 16 15:37 azkaban-exec-server-3.46.0.tar.gz-rw-r--r-- 1 yinzhengjie yinzhengjie 24333 Apr 16 15:18 azkaban-hadoop-security-plugin-3.46.0.tar.gz-rw-r--r-- 1 yinzhengjie yinzhengjie 349929173 Apr 16 15:11 azkaban-plugins-master.zip-rw-r--r-- 1 yinzhengjie yinzhengjie 23819895 Apr 16 15:18 azkaban-solo-server-3.46.0.tar.gz-rw-r--r-- 1 yinzhengjie yinzhengjie 20899667 Apr 16 15:37 azkaban-web-server-3.46.0.tar.gz-rw-r--r-- 1 yinzhengjie yinzhengjie 11194 Nov 30 2017 create-all-sql-3.46.0.sql[yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ mkdir /soft/azkaban[yinzhengjie@s101 data]$
2>.分别解压文件到/soft/azkaban中
[yinzhengjie@s101 data]$ tar -xzf azkaban-exec-server-3.46.0.tar.gz -C /soft/azkaban/[yinzhengjie@s101 data]$ tar -xzf azkaban-web-server-3.46.0.tar.gz -C /soft/azkaban/[yinzhengjie@s101 data]$
3>.进入到mysql命令行并创建数据库
[yinzhengjie@s101 data]$ mysql -uroot -pyinzhengjieWarning: Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 2Server version: 5.6.38 MySQL Community Server (GPL)Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> create database yinzhengjie_azkaban;Query OK, 1 row affected (0.00 sec)mysql> use yinzhengjie_azkaban;Database changedmysql> source /home/yinzhengjie/data/create-all-sql-3.46.0.sqlQuery OK, 0 rows affected (0.03 sec)Query OK, 0 rows affected (0.00 sec)Query OK, 0 rows affected (0.01 sec)Query OK, 0 rows affected (0.00 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.01 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.00 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.01 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.06 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.13 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.22 sec)Query OK, 0 rows affected (0.15 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.18 sec)Query OK, 0 rows affected (0.13 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.17 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.03 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.06 sec)Query OK, 0 rows affected (0.33 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.01 sec)Query OK, 0 rows affected (0.01 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.22 sec)Query OK, 0 rows affected (0.12 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.09 sec)Query OK, 0 rows affected (0.10 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.17 sec)Query OK, 0 rows affected (0.18 sec)Query OK, 0 rows affected (0.14 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.12 sec)Query OK, 0 rows affected (0.04 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.08 sec)Query OK, 0 rows affected (0.17 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.16 sec)Query OK, 0 rows affected (0.13 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.13 sec)Query OK, 0 rows affected (0.10 sec)Records: 0 Duplicates: 0 Warnings: 0Query OK, 0 rows affected (0.04 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected, 1 warning (0.00 sec)Query OK, 0 rows affected (0.19 sec)Query OK, 0 rows affected (0.23 sec)Query OK, 0 rows affected (0.08 sec)Query OK, 0 rows affected (0.13 sec)Query OK, 0 rows affected (0.01 sec)Query OK, 0 rows affected (0.06 sec)Query OK, 0 rows affected (0.19 sec)Query OK, 0 rows affected (0.27 sec)Query OK, 0 rows affected (0.18 sec)Query OK, 0 rows affected (0.16 sec)Query OK, 0 rows affected (0.17 sec)Query OK, 0 rows affected (0.00 sec)Query OK, 0 rows affected (0.19 sec)mysql> exit;Bye[yinzhengjie@s101 data]$
4>.创建SSL配置（完成下述工作后,将在当前目录生成 keystore 证书文件，需要输出初始密码，选择国家，在输入确认字母y其他按回车即可）
[yinzhengjie@s101 data]$ keytool -keystore keystore -alias jetty -genkey -keyalg RSAEnter keystore password: Re-enter new password: What is your first and last name? [Unknown]: What is the name of your organizational unit? [Unknown]: What is the name of your organization? [Unknown]: What is the name of your City or Locality? [Unknown]: What is the name of your State or Province? [Unknown]: What is the two-letter country code for this unit? [Unknown]: CNIs CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=CN correct? [no]: yEnter key password for(RETURN if same as keystore password): [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ ll | grep keystore-rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore[yinzhengjie@s101 data]$
5>.将keystore文件拷贝到web文件夹下
[yinzhengjie@s101 data]$ ll | grep keystore-rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore[yinzhengjie@s101 data]$ cp keystore /soft/azkaban/azkaban-web-server-3.46.0/[yinzhengjie@s101 data]$
6>.编辑配置文件【azkaban.properties】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties#Azkaban Personalization Settings#服务器UI名称,用于服务器上方显示的名字azkaban.name=yinzhengjie#描述azkaban.label=My Local Azkaban#UI颜色azkaban.color=#FF3601#指定首页路径azkaban.default.servlet.path=/index#默认根web目录web.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/#默认时区,已改为亚洲/上海 默认为美国default.timezone.id=Asia/Shanghai #Azkaban UserManager class#用户权限管理默认类user.manager.class=azkaban.user.XmlUserManager#用户配置,具体配置参加下文user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml#Loader for projects# global配置文件所在位置executor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.propertiesazkaban.project.dir=projects#数据库类型database.type=mysql#端口号mysql.port=3306#数据库连接IP mysql.host=s101#数据库实例名mysql.database=azkaban#数据库用户名 mysql.user=root#数据库密码 mysql.password=yinzhengjie#最大连接数 mysql.numconnections=100 # Velocity dev mode#关闭快速开发模式velocity.dev.mode=false# Jetty服务器属性.#最大线程数 jetty.maxThreads=25#Jetty SSL端口 jetty.ssl.port=8443#Jetty端口 jetty.port=8081#SSL文件名 jetty.keystore=keystore#SSL文件密码 jetty.password=yinzhengjie#Jetty主密码 与 keystore文件相同 jetty.keypassword=yinzhengjie#SSL文件名 jetty.truststore=keystore# SSL文件密码 jetty.trustpassword=yinzhengjie# 执行服务器属性#执行服务器端口executor.port=12321 # 邮件设置#发送邮箱mail.sender=xxxxxxxx@163.com#发送邮箱smtp地址 mail.host=smtp.163.com#发送邮件时显示的名称 mail.user=xxxxxxxx#邮箱密码 mail.password=**********#任务失败时发送邮件的地址 job.failure.email=xxxxxxxx@163.com#任务成功时发送邮件的地址 job.success.email=xxxxxxxx@163.comlockdown.create.projects=false#缓存目录 cache.directory=cache# JMX statsjetty.connector.stats=trueexecutor.connector.stats=true# Azkaban JobTypes 插件配置azkaban.jobtype.plugin.dir=plugins/jobtypes[yinzhengjie@s101 data]$
以上代码只是对该配置文件的一个详细说明，不建议在这个配置文件中写中文！在实际配置中，需要将这个文件拷贝两份，分别放在web和exec目录中，实际使用配置如下：
[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties# Azkaban Personalization Settingsazkaban.name=yinzhengjieazkaban.label=My Local Azkabanazkaban.color=#FF3601azkaban.default.servlet.path=/indexweb.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/default.timezone.id=Asia/Shanghai# Azkaban UserManager classuser.manager.class=azkaban.user.XmlUserManageruser.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml# Loader for projectsexecutor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.propertiesazkaban.project.dir=projectsdatabase.type=mysqlmysql.port=3306mysql.host=s101mysql.database=yinzhengjie_azkabanmysql.user=rootmysql.password=yinzhengjiemysql.numconnections=100# Velocity dev modevelocity.dev.mode=false# Azkaban Jetty server properties.jetty.use.ssl=falsejetty.maxThreads=25jetty.port=8081# Azkaban Executor settingsexecutor.port=12321# mail settingsmail.sender=mail.host=# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.# enduser -> myazkabanhost:443 -> proxy -> localhost:8081# when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.# azkaban.webserver.external_hostname=myazkabanhost.com# azkaban.webserver.external_ssl_port=443# azkaban.webserver.external_port=8081job.failure.email=job.success.email=lockdown.create.projects=falsecache.directory=cache# JMX statsjetty.connector.stats=trueexecutor.connector.stats=true# Azkaban plugin settingsazkaban.jobtype.plugin.dir=plugins/jobtypes[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.properties # Azkaban Personalization Settingsazkaban.name=yinzhengjieazkaban.label=My Local Azkabanazkaban.color=#FF3601azkaban.default.servlet.path=/indexweb.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/default.timezone.id=Asia/Shanghai# Azkaban UserManager classuser.manager.class=azkaban.user.XmlUserManageruser.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml# Loader for projectsexecutor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.propertiesazkaban.project.dir=projectsdatabase.type=mysqlmysql.port=3306mysql.host=s101mysql.database=yinzhengjie_azkabanmysql.user=rootmysql.password=yinzhengjiemysql.numconnections=100# Velocity dev modevelocity.dev.mode=false# Azkaban Jetty server properties.jetty.use.ssl=falsejetty.maxThreads=25jetty.port=8081# Azkaban Executor settingsexecutor.port=12321# mail settingsmail.sender=mail.host=# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.# enduser -> myazkabanhost:443 -> proxy -> localhost:8081# when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.# azkaban.webserver.external_hostname=myazkabanhost.com# azkaban.webserver.external_ssl_port=443# azkaban.webserver.external_port=8081job.failure.email=job.success.email=lockdown.create.projects=falsecache.directory=cache# JMX statsjetty.connector.stats=trueexecutor.connector.stats=true# Azkaban plugin settingsazkaban.jobtype.plugin.dir=plugins/jobtypes[yinzhengjie@s101 ~]$
7>.修改用户配置文件【azkaban-users.xml】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml[yinzhengjie@s101 data]$
8>.创建global.properties
[yinzhengjie@s101 data]$ touch /soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties[yinzhengjie@s101 data]$
9>.编写启动脚本
[yinzhengjie@s101 ~]$ more /usr/local/bin/azweb.sh #!/bin/bash#@author :yinzhengjie#blog:http://www.cnblogs.com/yinzhengjie#EMAIL:y1053419035@qq.com#判断用户是否传参if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exitfi#获取用户输入的命令cmd=$1case $cmd in start) cd /soft/azkaban /soft/azkaban/azkaban-web-server-3.46.0/bin/start-web.sh ;; stop) cd /soft/azkaban /soft/azkaban/azkaban-web-server-3.46.0/bin/shutdown-web.sh ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;;esacecho ========= azweb $cmd =============[yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ more /usr/local/bin/azexec.sh #!/bin/bash#@author :yinzhengjie#blog:http://www.cnblogs.com/yinzhengjie#EMAIL:y1053419035@qq.com#判断用户是否传参if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exitfi#获取用户输入的命令cmd=$1case $cmd in start) cd /soft/azkaban /soft/azkaban/azkaban-exec-server-3.46.0/bin/start-exec.sh ;; stop) cd /soft/azkaban /soft/azkaban/azkaban-exec-server-3.46.0/bin/shutdown-exec.sh ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;;esacecho ========= azexec $cmd =============[yinzhengjie@s101 ~]$ 10>.启动服务（启动web和executor）
[yinzhengjie@s101 ~]$ jps4970 Jps[yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ azweb.sh start========= azweb start =============[yinzhengjie@s101 ~]$ azexec.sh start========= azexec start =============[yinzhengjie@s101 ~]$ jps5024 AzkabanExecutorServer4993 AzkabanWebServer5044 Jps[yinzhengjie@s101 ~]$ netstat -untalp | grep 8081(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)tcp6 0 0 :::8081 :::* LISTEN 4993/java [yinzhengjie@s101 ~]$
11>.测试azkaban连接
输入用户名和密码信息：
登录成功会有以下页面：
二. Azkaban防坑小技巧
1>.job文件有空格
解决方案：
修改command.job配置文件，将里面多余的空格删除掉！然后重新上传job任务。
2>.Missing required property 'azkaban.native.lib'
解决方案：
[yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/[yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.propertiesazkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/[yinzhengjie@s101 ~]$
3>.
4>.