Using Ambari to Install DolphinScheduler: Super Simple!

Apache DolphinScheduler is a distributed, decentralized, and scalable visual DAG workflow task scheduling system. It is committed to solving the complex dependencies in data processing workflows, making the scheduling system ready to use.

Ambari is a tool for managing Hadoop clusters. Although Ambari does not directly support the installation of DolphinScheduler, we can integrate it into Ambari through a custom service. This article will describe how to use Ambari to install DolphinScheduler.

1. Ambari Service Installation Package

Place the service installation package that needs to be integrated into Ambari in the following directory:

/var/lib/ambari-server/resources/stacks/HDP/3.1/services

2. Restart the Ambari-server Service

sudo ambari-server restart

After the integration, you can proceed with the service installation on the Ambari web page.

1. Add Service and Select DolphinScheduler

2. DolphinScheduler Service Configuration Choose to use MySQL as the metadata database for the DolphinScheduler service.

3. Service Selection for Installation Select two DS-master nodes, at least two DS-ALERTS, DS-APIs, and deploy the remaining nodes as DS-worker nodes.

4. Error Handling for hue-desktop-ini If you have configured namenode-ha before installing DolphinScheduler, you may encounter errors related to ‘hue-desktop-ini’. You need to modify the params.py script.

Script location: /var/lib/ambari-server/resources/stacks/HDP/3.3/services/DOLPHINSCHEDULER/package/scripts/params.py
Modify the content:
# hdfs_httpfs_host = config['configurations']['hue-desktop-ini']['hdfs_httpfs_host']
hdfs_httpfs_host = "dfs_ha_enabled"  # Comment out the original line and assign any value to hdfs_httpfs_host

After making the changes, save and restart the ambari-server service to reinstall DolphinScheduler.

5. Copy MySQL Driver to All Nodes’ libs Directory

cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/alert-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/api-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/master-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/worker-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/3.3.1.0-004/dolphinscheduler/tools/libs/

6. Change Ownership of All Services to hdfs User

chown -R hdfs:hdfs /var/log/dolphinscheduler
chown -R hdfs:hdfs /usr/hdp/3.3.1.0-004/dolphinscheduler
chown -R hdfs:hdfs /etc/dolphinscheduler/
ln -s /usr/hdp/3.1.4.0-315/dolphinscheduler dolphinscheduler

After making these changes, start DolphinScheduler from the Ambari management platform. The metadata will be automatically initialized and tables will be created during the startup process.

Join the Community

There are many ways to participate and contribute to the DolphinScheduler community, including:

Documents, translation, Q&A, tests, codes, articles, keynote speeches, etc.

We assume the first PR (document, code) to contribute to be simple and should be used to familiarize yourself with the submission process and community collaboration style.

So the community has compiled the following list of issues suitable for novices: https://github.com/apache/dolphinscheduler/contribute

• List of non-newbie issues:

https://github.com/apache/dolphinscheduler/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+

• How to contribute:

Https://github.com/apache/dolphinscheduler/blob/8944fdc62295883b0fa46b137ba8aee4fde9711a/docs/docs/en/contribute/join/contribute.md

• GitHub Code Repository: https://github.com/apache/dolphinscheduler
• Official Website:https://dolphinscheduler.apache.org/en-us
• Mail List:dev@dolphinscheduler@apache.org
• X.com:@DolphinSchedule
• YouTube:https://www.youtube.com/@apachedolphinscheduler
• Slack:https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-1cmrxsio1-nJHxRJa44jfkrNL_Nsy9Qg
• Contributor Guide:https://dolphinscheduler.apache.org/en-us/community

Your Star for the project is essential, don’t hesitate to lighten a Star for Apache DolphinScheduler ❤️