Updated on September 10, 2024 in #flask

Inspect the Stats and Get a Status Report of Your Celery Workers

You can use this to confirm your workers are configured how you'd like and see what tasks they are executing.

Celery’s CLI has a number of useful commands to get details about your workers.

I explored this recently while tweaking a few worker settings, such as setting the number of max tasks that a worker can execute before that worker gets replaced with a new one as well the max memory a worker can use. Both can help combat memory leaks.

I wanted peace of mind to know that the config settings I applied were really applied.

# A Few Useful Commands

All of these commands were run against my example Flask / Docker starter web app.

The ./run cmd shortcut you’ll see below is available in that project to exec into the web container and run any command (in this case celery). In your app’s case you’ll want to replace the -A flag with the path to your Celery app.

Ping Pong

One of the most basic commands to check if things are working is to ping your worker and see if it replies back. You can use this to verify your connection URLs are set up correctly.

$ ./run cmd celery -A "hello.app.celery_app" inspect ping
->  celery@99eb7233e1d4: OK
        pong

1 node online.

Verify Configuration

This one is useful to make sure everything is configured in an expected way based on the config settings you pass into Celery. The example app doesn’t do a lot of advanced configuration but your settings will be listed near the bottom.

$ run cmd celery -A "hello.app.celery_app" report

software -> celery:5.4.0 (opalescent) kombu:5.4.0 py:3.12.5
            billiard:4.2.0 redis:5.0.8
platform -> system:Linux arch:64bit
            kernel version:5.15.153.1-microsoft-standard-WSL2 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://redis:6379/0

deprecated_settings: None
broker_url: 'redis://redis:6379/0'
result_backend: 'redis://redis:6379/0'
include: []

Detailed Stats

Sometimes it’s handy to know how long a worker has been up for or how many processes are running. You can find out these details and more with the inspect stats command.

$ run cmd celery -A "hello.app.celery_app" inspect stats
->  celery@99eb7233e1d4: OK
    {
        "broker": {
            "alternates": [],
            "connect_timeout": 4,
            "failover_strategy": "round-robin",
            "heartbeat": 120.0,
            "hostname": "redis",
            "insist": false,
            "login_method": null,
            "port": 6379,
            "ssl": false,
            "transport": "redis",
            "transport_options": {},
            "uri_prefix": null,
            "userid": null,
            "virtual_host": "0"
        },
        "clock": "310",
        "pid": 1,
        "pool": {
            "implementation": "celery.concurrency.prefork:TaskPool",
            "max-concurrency": 4,
            "max-tasks-per-child": "N/A",
            "processes": [
                9,
                10,
                11,
                12
            ],
            "put-guarded-by-semaphore": false,
            "timeouts": [
                0,
                0
            ],
            "writes": {
                "all": "",
                "avg": "0.00",
                "inqueues": {
                    "active": 0,
                    "total": 4
                },
                "raw": "",
                "strategy": "fair",
                "total": 0
            }
        },
        "prefetch_count": 16,
        "rusage": {
            "idrss": 0,
            "inblock": 272,
            "isrss": 0,
            "ixrss": 0,
            "majflt": 1,
            "maxrss": 85564,
            "minflt": 54782,
            "msgrcv": 0,
            "msgsnd": 0,
            "nivcsw": 251,
            "nsignals": 0,
            "nswap": 0,
            "nvcsw": 1074,
            "oublock": 16,
            "stime": 0.244528,
            "utime": 2.750949
        },
        "total": {},
        "uptime": 306
    }

1 node online.

Are We Busy?

You can also check to see if any jobs are actively running. You can use this to help detect if there are any stuck jobs being processed.

Empty means your queue is empty and there’s no active jobs being processed.

$ run cmd celery -A "hello.app.celery_app" inspect active
->  celery@99eb7233e1d4: OK
    - empty -

1 node online.

There’s quite a few other commands you can run too, feel free to check out the help menu.

Flower

Besides the CLI, there is a web based Celery monitoring tool called Flower which will show you all sorts of details about your workers and tasks.

I’ve used it in the past but I tend not to include it in most projects which is why it’s not in the starter app. However, I wanted to bring it up in case it could be useful for you.

With Docker Compose it’s easy to set up. It’s just another service that you run, but be careful to avoid making it publicly available on the internet if you publish its web port and aren’t using a firewall to restrict ports.

The video below runs the above commands within the project.

# Demo Video

Timestamps

• 0:36 – The example app
• 1:30 – Pinging your workers
• 1:56 – Verifying your Celery config
• 2:47 – Defining a few custom settings
• 4:26 – Confirming our new settings are defined
• 5:03 – Checking the stats and uptime of your workers
• 5:48 – Looking at active jobs
• 6:28 – Other Celery CLI commands
• 6:51 – Flower is a web UI to get details about your jobs

Do you use Celery on a regular basis? What commands do you use?