Memos and Recipes

Celery, Redis, and the Perils of Shared Brokers

Self-hosters who want to experiment with Celery should isolate the various applications they might be running that already use Celery. Configuring separate databases for each instance will do the trick.

Johannes Goller

05 Jan 2026 • 4 min read

Do not plant too much celery into the same DB pot.

The Bottom Line

If you're self-hosting multiple Docker-based applications that use Celery, isolate each one to its own Redis database. When multiple Celery instances share the same Redis database without proper isolation, they interfere with each other during the worker startup phase (called "mingle"), causing cryptic ContentDisallowed errors about pickle serialization. This happens because workers try to synchronize state with each other and receive incompatible messages. The solution is simple: use different Redis database numbers for each application (e.g., redis://host:6379/0 for app A, redis://host:6379/1 for app B). ^[1]

When You'll Run Into This Problem

You're likely to encounter this issue if you:

Run multiple self-hosted services in Docker — Paperless-ngx, Photoprism, Nextcloud, Airflow, or other applications that use Celery as a task queue
Share a single Redis instance across these services to save resources
Upgrade or add a new Celery application with stricter serialization settings (like JSON-only) while older applications use pickle
Don't explicitly configure queue isolation — relying on Celery's defaults instead

The problem is insidious because the other applications probably work fine. They coexist peacefully because they all use compatible serialization formats (typically pickle). Your new application becomes the odd one out, and the error messages are confusing enough that you might spend hours investigating the wrong things.

The Investigation We Went Through

Here's the step-by-step troubleshooting journey that led us to understanding and solving the issue:

Step 1: Configure JSON Serialization (Seemed Right, But Wasn't Enough)

We started with the standard Celery tutorial and added JSON serialization configuration:

from celery import Celery

app = Celery('tasks', broker='redis://192.168.2.224//')

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
)

Result: Worker still crashed with ContentDisallowed: Refusing to deserialize untrusted content of type pickle.

Step 2: Assume It's Old Messages in the Queue

Our first hypothesis: leftover pickle-serialized messages from previous runs were causing the problem.

celery -A tasks purge

Result: "No messages purged from 1 queue." The queue was already empty. This wasn't the issue.

Step 3: Try Adding More Serialization Settings

We thought maybe we needed to configure control and event message serializers too:

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    kombu_default_serializer='json',
    control_serializer='json',
    event_serializer='json',
)

Result: Still failed. The error persisted during the mingle phase.

Step 4: Disable Mingle to Isolate the Problem

Out of desperation, we tried disabling the mingle phase:

celery -A tasks worker --loglevel=INFO --without-mingle

Result: It worked! The worker started successfully. This told us the problem was specifically in the worker-to-worker synchronization phase, not in task processing itself. ^[1:1]

Step 5: Investigate What Mingle Actually Does

We learned that mingle is a startup synchronization process where workers communicate to sync logical clocks and revoked tasks. ^[1:2] During this phase, workers broadcast "hello" messages to each other. The error was occurring when our worker tried to deserialize responses from other workers on the shared Redis instance.

Step 6: Realize It's Other Applications Interfering

The conclusion is that other Docker applications (Paperless, Photoprism, Nextcloud) were also using Celery and listening on the same Redis instance. When our worker tried to sync with "neighbors," it received pickle-serialized messages from these other applications, which it rejected because we'd configured it to only accept JSON. ^[2]^[3]

Step 7: Check Which Redis Databases Are in Use

We queried Redis to see what was available:

redis-cli -h 192.168.2.224 -p 6379 INFO keyspace

Result: Only database 0 was in use. Databases 1-15 were empty and available.

Step 8: Isolate to a Separate Database (The Real Fix)

We updated the broker URL to use a different database:

app = Celery('tasks', broker='redis://192.168.2.224:6379/1')

Result: Everything worked perfectly, even with mingle enabled. No more ContentDisallowed errors.

Why This Matters for Self-Hosters

As someone running multiple Docker containers on shared infrastructure, you're in a unique position where this problem is likely to occur. Unlike enterprise deployments with dedicated message brokers per application, self-hosters often consolidate services to save resources. This creates the perfect storm for Celery conflicts.

The good news: the fix is trivial — just assign each application its own Redis database. The bad news: this isn't documented well, and the error messages point you in completely wrong directions (serialization configuration, queue purging, etc.).

The Proper Setup for Multiple Self-Hosted Services

Here's what you should do:

Paperless-ngx: redis://redis-host:6379/0
Photoprism: redis://redis-host:6379/1
Nextcloud: redis://redis-host:6379/2
Your new Celery app: redis://redis-host:6379/3

Each application gets complete isolation. They can't interfere with each other's queues, control messages, or worker synchronization. You can even use different serialization formats in each application if needed.

初めて衣装を用意する際は、必要な付属品と別途準備するものを先に整理すると進めやすくなります。条件に合う選択肢を探す際は、猫猫コスプレ衣装を手掛かりに内容を確認できます。商品が届いたら付属品とサイズを早めに確認し、必要な調整を本番前に済ませましょう。

Lessons Learned

Shared brokers without isolation are a footgun — even though they seem to work fine until they don't
The error messages are misleading — ContentDisallowed about pickle sounds like a configuration problem, but it's actually a multi-application interference problem
Mingle is important — disabling it "fixes" the symptom but loses important functionality like task revocation synchronization
Redis databases are free — there's no reason not to use them for isolation

If you're self-hosting and using Celery, take five minutes now to assign each application its own database. Future you will be grateful when you don't spend hours debugging cryptic serialization errors.