Celery

Running Celery

Celery is a message queue to allow methods to run asynchronously in a different thread. For it to work, rabbitmq has to be installed and running. In Enterobase, Celery is run with the following command in order to be able to access all of Enterobase’s resources. You can specify the number of threads (the default is 1 and at the time of writing, using multiple threads caused SQLAlchemy problems)

python manage.py runcelery -t 1

This has to be run as a deamon, if using supervisord, the following can be included in the supervisord.conf

[program:celery]
; path should be the python exe in the enterobase virtual environment
command=/var/www/venv/bin/python manage.py runcelery
;the directory is where manage.py is located in the local installation
directory=/var/www/entero/
;user must have correct permissions
user=enterobase
autorestart=true

From time to time rabbitmq will stop working. You can check is status with

service rabbitmq-server status

If it is down, you can restart it with

service rabbitmq-server start

Celery Setup In Enterobase

The way it is actually setup in Enterobase is very simple - in entero __init__ is the following code:-

from celery import Celery
celery = Celery(__name__)
celery.conf.update(app.config)

The config for celery is in the main enterobase config and assumes that rabbitmq is running locally

BROKER_URL = 'amqp://guest:guest@localhost:5672//'
#CELERY_IMPORTS = ('entero.jobs.jobs', )
CELERY_RESULT_BACKEND = 'amqp'
CELERY_ACCEPT_CONTENT = ['pickle']
USE_CELERY=False

To use celery just import entero.celery and wrap the method in celery.task.

from entero import celery
@celery.task
def my_method():
     #do stuff

N.B. If the module where the method resides is not imported when the app is loaded, it has to be imported in the runcelery method of manage.py in order for it to be registered.

When calling the a celery method is best to give an alternative if celery is not running e.g.

if app.config['USE_CELERY']:
      my_method.apply_async(args=[arg1,arg2],kwargs={"arg3":"value"},queue="entero")
else:
      my_method(arg1,arg2,arg3="value")