Golden Blog

Golden Blog

0%

Ambari编译安装

发表于 更新于 分类于 ， 阅读次数： Valine：

本文主要包括：

  • Ambari编译
  • Ambari安装

Apache Ambari是一个用于配置、管理和监视ApacheHadoop集群的工具。Ambari由一组RESTful API和基于浏览器的管理界面组成。
Ambari使系统管理员能够：

  1. 配置Hadoop群集
  2. Ambari提供了一个易于使用的分步向导，用于在任意数量的主机上安装Hadoop服务。
  3. Ambari负责集群Hadoop服务的配置。
  4. 管理Hadoop群集
  5. Ambari提供了在整个集群中启动、停止和重新配置Hadoop服务的中央管理。
  6. 监视Hadoop群集
  7. Ambari提供了一个用于监控Hadoop集群运行状况和状态的仪表板。
  8. Ambari将在您需要注意时发送电子邮件（例如，节点故障、剩余磁盘空间不足等）。

Ambari编译

  • 下载源码
    wget https://www-eu.apache.org/dist/ambari/ambari-2.7.7/apache-ambari-2.7.7-src.tar.gz (use the suggested mirror from above)
tar xfvz apache-ambari-2.7.7-src.tar.gz
cd apache-ambari-2.7.7-src
mvn versions:set -DnewVersion=2.7.7.0.0
 
pushd ambari-metrics
mvn versions:set -DnewVersion=2.7.7.0.0
popd

    安装必要的软件

  1. java 1.8
  2. maven-3.6.3
  3. node <= v11.15.0
  4. brunch
    # Find the node root directory
cd `npm root -g`

# Change Taobao source
npm config set registry https://registry.npm.taobao.org
npm info underscore

# Install brunch-1.7.20
npm install -g brunch@1.7.20

# View the installed version
brunch -V
  5. yum install -y rpm-build gcc-c++ python-devel git
  6. 安装yarn：
    npm install -g yarn # 这里要开外网

    更换镜像源

    修改.m2/settings.xml
    <mirrors>
  <!-- Amabri build mirror source -->
	<mirror>
		<id>alimaven</id>
		<name>aliyun maven</name>
		<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
		<mirrorOf>central</mirrorOf>
	</mirror>
	<mirror>
		<id>alimaven</id>
		<mirrorOf>central</mirrorOf>
		<name>aliyun maven</name>
		<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
	</mirror>
	<mirror>
		<id>nexus-hortonworks</id>
		<mirrorOf>*,!central</mirrorOf>
		<name>Nexus hortonworks</name>
		<url>https://repo.hortonworks.com/content/groups/public/</url>
	</mirror>
	<mirror>
		<id>HDPReleases</id>
		<name>HDP Releases</name>
		<url>http://repo.hortonworks.com/content/repositories/releases/</url>
		<mirrorOf>central</mirrorOf>
	</mirror>
</mirrors>

    配置github.com

    vim /etc/hosts
140.82.113.4 github.com

    修改 ambari-admin/pom.xml

    # 修改105
vim ambari/ambari-admin/pom.xml

# 修改前 和 修改后
<argument>${basedir}/src/main/resources/ui/admin-web/node_modules/bower/bin/bower</argument>
<argument>bower</argument>

    修改依赖地址

    修改 ambari-2.7.4/ambari-metrics/pom.xml 文件，依赖包修改为本地下载好的位置
    <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <python.ver>python &gt;= 2.6</python.ver>
  <deb.python.ver>python (&gt;= 2.6)</deb.python.ver>
  <!--TODO change to HDP URL-->   
  <hbase.tar>file:///opt/soft/hbase-2.1.0-bin.tar.gz</hbase.tar>
  <hbase.folder>hbase-2.1.0</hbase.folder>
  <hadoop.tar>file:///opt/soft/hadoop-3.1.1.tar.gz</hadoop.tar>
  <hadoop.folder>hadoop-3.1.1</hadoop.folder>
  <hadoop.version>3.1.1</hadoop.version>
  <grafana.folder>grafana-10.0.0</grafana.folder>
  <grafana.tar>file:///opt/soft/grafana-enterprise-10.0.0.linux-amd64.tar.gz</grafana.tar>
  <phoenix.tar>file:///opt/soft/phoenix-hbase-2.1-5.1.0-bin.tar.gz</phoenix.tar>
  <phoenix.folder>phoenix-hbase-2.1-5.1.0-bin</phoenix.folder>
  <resmonitor.install.dir>/usr/lib/python2.6/site-packages/resource_monitoring</resmonitor.install.dir>
  <powermock.version>1.6.2</powermock.version>
  <fasterxml.jackson.version>2.10.0</fasterxml.jackson.version>
  <fasterxml.jackson.databind.version>2.10.0</fasterxml.jackson.databind.version>
</properties>
    修改ambari/ambari-metrics/ambari-metrics-timelineservice/pom.xml
    <move
    file="${project.build.directory}/embedded/${phoenix.folder}/phoenix-server-hbase-2.1-5.1.0.jar"
    tofile="${project.build.directory}/embedded/${hbase.folder}/lib/phoenix-server-hbase-2.1-5.1.0.jar"
/>

    修改多处zookeeper的版本以及ambari-utility版本

修改多处源码pom文件

ambari源码内多处pom文件写死了repository和mirror，里面的很多url都是访问不了的。
把这些都删除掉。

可以参考Ambari源码编译、安装（包括异常处理）
Ambari官网

遇到问题

  1. Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:install-node-and-yarn
    [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:install-node-and-yarn (install node and yarn) on project ambari-web: Could not download Yarn: Could not download https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz: Remote host closed connection during handshake: SSL peer shut down incorrectly -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :ambari-web
    解决办法：
    mkdir -p ~/.m2/repository/com/github/eirslett/yarn/0.21.3/yarn-0.21.3./
cd ~/.m2/repository/com/github/eirslett/yarn/0.21.3/yarn-0.21.3./
wget https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz
  2. Failed to run task: 'yarn install --ignore-engines --pure-lockfile' failed
    [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :ambari-admin
    解决办法：安装git，并且调整nodejs版本<= v11.15.0

  1. ambari-metrics-storm-sink-legacy: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink-legacy:jar:2.7.7.0.0

    [ERROR] Failed to execute goal on project ambari-metrics-storm-sink-legacy: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink-legacy:jar:2.7.7.0.0: Could not find artifact org.apache.storm:storm-core:jar:0.10.0.2.3.0.0-2557 in apache-hadoop (https://repo.hortonworks.com/content/groups/public/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :ambari-metrics-storm-sink-legacy

    解决办法： 进入ambari-metrics，修改pom.xml
    把第一个 repository 替换为 http://nexus-private.hortonworks.com/nexus/content/groups/public

  2. 下载hbase/hadoop/grafana/phoenix很慢
    解决办法：提前使用wget下载下来，并在ambari-metrics/ambari-metrics-timelineservice，ambari-metrics/ambari-metrics-assembly,ambari-metrics/ambari-metrics-grafana修改pom文件

    <!--
 <get
      src="${phoenix.tar}"
      dest="${project.build.directory}/embedded/phoenix.tar.gz"
      usetimestamp="true"
  />
-->
 <copy todir = "${project.build.directory}/embedded/"
       file = "/home/gujc/phoenix.tar.gz"
  />

具体可以参考
国内网络编译，Ambari 2.7.6 全部模块源码编译笔记
自建云服务器集群离线安装Ambari2.7.5和HDP3.1.5(一)
自建云服务器集群离线安装Ambari2.7.5和HDP3.1.5(二)

Ambari 安装

数据库初始化

CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari@123';
GRANT ALL ON ambari.* TO 'ambari'@'%';
-- souce这个步骤需要在ambari-server setup之后操作
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
echo server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar >> /etc/ambari-server/conf/ambari.properties

Ambari Server 安装

[root@VM-4-14-centos ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? n
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Invalid characters in password. Use only alphanumeric or _ or - characters
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Enter full path to custom jdbc driver: /opt/soft/mysql-connector/mysql-connector-java-8.0.28.jar
Copying /opt/soft/mysql-connector/mysql-connector-java-8.0.28.jar to /usr/share/java
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.7.0.0.jar

Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

查看 Ambari 服务器日志：tail -f /var/log/ambari-server/ambari-server.log
要访问 Ambari，请转到:http://{ambari-server-hostname}:8080
使用用户名admin 和密码 admin登录

Ambari agent安装

在集群中的每台主机上手动安装并启动 Ambari 代理
安装 Ambari 代理 RPM，在 RHEL/CentOS 上：
yum install ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm
通过编辑主机名行，在 /etc/ambari-agent/conf/ambari-agent.ini 中编辑 Ambari 服务器的位置。
启动 Ambari 代理：ambari-agent start
查看 Ambari 代理日志：tail -f /var/log/ambari-agent/ambari-agent.log

遇到问题：

  1. Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property “server.jdbc.driver.path=[path/to/custom_jdbc_driver]” in ambari.properties.
    解决办法：
    将mysql的驱动包放到/usr/share/java
在/etc/ambari-server/conf/ambari.properties中添加
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
  2. Table 'ambari.metainfo' doesn't exist
    Waiting for server start...........ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
# 在/var/log/ambari-server/ambari-server.log报错：
Caused by: java.sql.SQLSyntaxErrorException: Table 'ambari.metainfo' doesn't exist
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1009)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:1009)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:644)
    解决办法：source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
  1. fsdf
  2. fsdf
  3. dsfas
  4. dfasdf
  5. dfasdf

安装Apache Hadoop等组件

网上的Ambari文档大部分都是直接安装HDP版本的软件，但是，HDP目前不开源，所以公司需要使用完全开源的大数据组件
要想使用Amabri安装Apache版本的相应组件，安装步骤应该如下：

  1. 使用Ambari安装HDP版本的rpm包（为了让它创建集群环境，一些service目录等）
  2. 替换已经安装的组件，替换成tar.gz，这种方式的好处是，各个组件解耦，并且不需要编译
  3. 测试替换后组件的稳定性

下面首先安装正常HDP的rpm包，网上很多资源。这里不赘述