Data Engineering Series | Run Apache-Airflow as a Service on Centos 7 | Apache Airflow #2

Image by Ben Jessop

Centos has a built-in mechanism to create custom services, enabling them to get started at system boot time and start/stop them as a service. In this article, I am going to share a simple way to create a service wrapper for your apache-airflow web server so you can run it as a service.

Create a Service for Airflow Webserver

1. Create a Service

First, create a file named airflow-webserver.service using the below command

sudo touch /etc/systemd/system/airflow-webserver.service

2. Write Service Configuration

Now, open the airflow-webserver.service file using vim editor and paste the below lines in it.

EnviromentFile and ExecStart depending on the virtual environment and airflow installation location.

[Unit]
Description=Airflow webserver daemon
After=network.target mysql.service
Wants=mysql.service
[Service]
EnvironmentFile=/home/user/airflow/.env
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/bin/bash -c 'source  /home/user/All_ENV/airflow_venv/bin/activate ; airflow webserver --pid /home/user/airflow/airflow-webserver.pid'
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target

You may be wondering what all these directives do, let me describe a few of these for you:

Description : As the name suggestes it is a small description about      the service
After :It simply means that your service must be started after the services that are required for your deamon to run, like for apache-airflow we need network, mysql to be ready before it gets started. If your apache-airflow installation requires redis or rabbitmq services than you can specify the dependency here
EnvironmentFile : it specifies the file where service will find its environment variables
User : set your actual username here, the specified userid will be used to invoke the service
ExecStart : here you can specify the command to be executed or set the proper path to your script that needs to be excuted. in our case we simple need to run the command airflow webserver
Restart : by default, systemd does not restart your service if the program exits for whatever reason. This is usually not what you want for a service that must be always available, so we’re instructing it to always restart on-failure
RestartSec : by default, systemd attempts a restart after 100ms. You can specify the number of seconds to wait before attempting a restart, using RestartSec.

3. Load Enable and Start the Service

start load the service first by using below command:

sudo systemctl daemon-reload

Now enable the service airflow-webserver.service

sudo systemctl enable airflow-webserver.service

and you can simply start the service by:

sudo systemctl start airflow-webserver.service

4. Check the Service

You can check the status of service by :

sudo systemctl status airflow-webserver.service

5. Stop the Service

To stop the service use following command

sudo systemctl stop airflow-webserver.service

Create a Service for Airflow Scheduler

In order to create a service for the scheduler, the process is same, you just need to make another service entry by creating a file

sudo touch /etc/systemd/system/airflow-scheduler.service

Now, paste the below configuration code inside it

[Unit]
Description=Airflow webserver daemon
After=network.target mysql.service
Wants=mysql.service
[Service]
EnvironmentFile=/home/user/airflow/.env
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/bin/bash -c 'airflow scheduler'
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target

Thanks….