Celery worker as a daemon on AWS ec2 ubuntu Instance

You probably want to use a daemonization tool to start the worker in the background. Running the worker in the background as a daemon see Daemonization for more information.

Create a configuration file

Access your AWS Ubuntu instance on terminal

ssh ubuntu@your-aws-instance-public-ip -i key.pem

Create the empty /etc/default/celeryd for the init script

sudo touch /etc/default/celeryd
sudo vim /etc/default/celeryd

Copy below and paste to /etc/default/celeryd

Here Project_name is your Django app and folder_name is your repo name.

# Names of nodes to start
# most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10
# Absolute or relative path to the 'celery' command:
# which celery
CELERY_BIN="/usr/local/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="<Project_name>"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# Where to chdir at start.
CELERYD_CHDIR="/home/ubuntu/<folder_name>/"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"
# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="ubuntu"
CELERYD_GROUP="ubuntu"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

Create the init script

Create the init script in /etc/init.d/celeryd. See the extra/generic-init.d/ directory Celery.

# download
$ wget https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd
# executable
$ chmod +x celeryd
# move
$ sudo mv celeryd /etc/init.d/

Create logs directory

# make new dir
$ sudo mkdir /var/log/celery
# change owner
$ sudo chown -R ubuntu: /var/log/celery

Create the empty /etc/default/celertbeat for the init script

sudo touch /etc/default/celerybeat
sudo vim /etc/default/celerybeat

Copy below and paste to /etc/default/celerybeat

Here Project_name is your Django app and folder_name is your repo name.

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="<Project_name>"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# Where to chdir at start.
CELERYBEAT_CHDIR="/home/ubuntu/<folder_name>/"
# Extra arguments to celerybeat
CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule"

Create the init script

Create the init script in /etc/init.d/celerybeat. See the extra/generic-init.d/ directory Celery.

# download
$ wget https://github.com/celery/celery/blob/master/extra/generic-init.d/celerybeat
# executable
$ chmod +x celerybeat
# move
$ sudo mv celerybeat /etc/init.d/

You should try running them in verbose mode

sudo sh -x /etc/init.d/celeryd start
sudo sh -x /etc/init.d/celerybeat start

Restart celeryd & celerybeat

sudo /etc/init.d/celeryd restart
sudo /etc/init.d/celerybeat restart

Thank you guys, Hope my article helps you :)