Databases

Django ManyToManyField Missing Table: Docker Postgres Fix

Fix Django ManyToManyField join table missing in Docker Postgres despite migrations applied. Troubleshoot ProgrammingError, sync django_migrations table, Docker pitfalls, and best practices for reliable migrations.

1 answer• 1 view

Django ManyToMany join table missing despite migrations applied (Docker + Postgres)

Problem

In a Django project using PostgreSQL in Docker, a ManyToManyField was added to a model. Migrations are ignored in .gitignore, model changes were pushed to GitHub, pulled on the server, and docker compose up ran makemigrations and migrate without errors. The migration file includes the ManyToManyField, and Django shows it as applied. However, accessing the relation at runtime raises a ProgrammingError because the join table does not exist.

Environment

  • Django
  • PostgreSQL
  • Docker

Model

python
class Contact(models.Model):
 # ...

class CallCampaign(models.Model): 
 user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) 
 error = models.TextField(blank=True, null=True) 
 company = models.ForeignKey(Company, on_delete=models.CASCADE) 
 selected_contacts = models.ManyToManyField('Contact', blank=True) 
 # ...

Migration Snippet

python
migrations.CreateModel( 
 name='CallCampaign', 
 fields=[ 
 ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 
 ('error', models.TextField(blank=True, null=True)),
 ('company', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='companies.company')), 
 ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), 
 ('selected_contacts', models.ManyToManyField(blank=True, to='contacts.contact')), 
 ], 
),

Migration marked as applied in django_migrations.

Runtime Error

python
>>> from contacts.models import CallCampaign 
>>> c = CallCampaign.objects.first() 
>>> c.selected_contacts.all()

Error:

django.db.utils.ProgrammingError: relation "contacts_callcampaign_selected_contacts" does not exist
LINE 1: ..."."created_at" FROM "contacts_contact" INNER JOIN "contacts_...

Database Tables

Existing: contacts_callcampaign, contacts_contact, contacts_callhistory

Missing: contacts_callcampaign_selected_contacts

Commands

bash
python manage.py makemigrations # No changes detected
python manage.py migrate # No migrations to apply

Docker Setup (simplified)

yaml
services: 
 db: 
 image: postgres:15 
 volumes: 
 - postgres_data:/var/lib/postgresql/data/ 
 # healthcheck...
 web: 
 build: . 
 command: > 
 sh -c " 
 python manage.py makemigrations --noinput && 
 python manage.py migrate --noinput
 " 
 depends_on: 
 db: 
 condition: service_healthy 
volumes: 
 postgres_data:

Persistent volume used.

Questions

  1. Why is the migration marked as applied without creating the ManyToMany join table?
  2. Is this expected Django behavior (lack of atomicity)?
  3. How to prevent this in Docker with persistent Postgres databases?
  4. What to do now to fix it?

Seeking best practices for Django migrations with ManyToManyFields in Docker/Postgres setups.

Django ManyToManyField join tables often vanish in Docker Django PostgreSQL environments despite migrations applied because Django’s migration tracker (django_migrations table) gets out of sync with the actual schema—especially after Git-pulled model changes without migration files committed. Silent failures during migrate (due to app dependencies, Postgres strictness, or Docker volume quirks) mark the migration as done without creating the table like contacts_callcampaign_selected_contacts. Here’s how to diagnose, fix it safely, and prevent repeats with best practices for docker django postgresql setups.

Contents

Sources

  1. How to Fix Django DatabaseError: No such table
  2. Django Ticket #24524: Automatic migrations prevent creation
  3. Django 1.8 migrate is not creating tables - Stack Overflow
  4. Django migrate doesn’t create tables - Stack Overflow
  5. Django and Postgres inserting into many-to-many table - Stack Overflow
  6. Django Ticket #3779: Problem with ManyToManyFields and syncdb
  7. Migrations | Django documentation
  8. Django doesn’t create any tables in Postgres - Stack Overflow
  9. Django Migration Error: Column does not exist - Stack Overflow
  10. Django model changes not reflected on Postgres Docker - Stack Overflow
  11. Django migrations failing in Docker after adding model - Stack Overflow
  12. Django migrations issues with Postgres - Stack Overflow
  13. How to Fix ‘Relation Does Not Exist’ Error in Django

Conclusion

The root of your missing contacts_callcampaign_selected_contacts table boils down to Django’s optimistic migration tracking clashing with Docker’s rebuild nature and Postgres’s no-nonsense schema rules—migrations django gets marked applied, but the work skips silently. Quick fixes like faking or cleaning django_migrations often resolve it without data loss, but commit those migration files to Git religiously for docker django postgresql sanity. Adopt the best practices here: volume-check your Docker setup, sequence commands tightly, and test migrations locally first. You’ll dodge this ProgrammingError headache for good, keeping your ManyToManyField relations rock-solid even under container restarts.

Why the Join Table is Missing Despite Migrations Applied

Ever pulled code on a server, run docker compose up, and watched migrate yawn “No migrations to apply”—only for c.selected_contacts.all() to explode with “relation does not exist”? You’re not alone. Django auto-generates ManyToManyField join tables (like contacts_callcampaign_selected_contacts) during the CreateModel migration that includes your selected_contacts field. But here’s the kicker: the django_migrations table logs it as applied without guaranteeing the schema change stuck.

Why? Django assumes atomicity per migration, but real-world hiccups break it. Official Django docs on migrations warn that apps without synced migrations can’t reliably handle relations like ManyToManyField to migrated apps. Your snippet shows the field in CreateModel—great—but if dependencies (say, contacts app before callcampaigns) misfire, Postgres balks at foreign keys. No error during migrate, but boom at runtime.

And Docker? It amplifies this. Persistent volumes keep your Postgres data, but if the container rebuilds without all migration files (ignored in .gitignore—huge red flag), Django thinks everything’s current based on django_migrations. Result: table ghosted.

Short bursts fix sentiment: frustrating. But predictable once you spot the patterns.

Is This Expected Django Behavior?

Not “expected,” but common enough to feel standard in docker django setups. Migrations aren’t always atomic across multi-app relations—Django ticket #24524 nails it: unmigrated apps referencing migrated ones fail eventually. Postgres enforces this strictly, unlike SQLite’s leniency. Your case? Migration ran, logged success, but the JOIN table creation silently skipped due to order or connection blips.

Docker Postgres Pitfalls with Django Migrations

Docker django postgresql combos are migration minefields. Your setup looks solid—postgres_data volume, healthcheck, depends_on—but subtle gremlins lurk.

First, .gitignore ignoring migrations? Disaster waiting. Pulled models trigger makemigrations, but without committed history, Django sees “no changes” against a stale django_migrations. Stack Overflow thread on Docker model changes echoes this: volumes persist DB, but Django queries the wrong state if migrations aren’t baked in.

Postgres strictness bites too. Ticket #3779 explains: if models load reverse-order, FKs in ManyToMany through-tables fail creation. Your contacts.Contact and CallCampaign cross-apps? Prime suspect.

Network flakiness? Django Forum post shows Docker DNS resolving “postgres” failing mid-migrate, leaving partial schemas. Commands report success; tables don’t.

What about rebuilds? Another SO post matches: new models, lost migrations, table conflicts. Your persistent volume saves data but not sync.

Frustrated yet? Good—now let’s diagnose.

Step-by-Step Diagnosis

Don’t nuke yet. Verify first.

  1. Check django_migrations: Docker exec into web container, python manage.py dbshell, then:
SELECT * FROM django_migrations WHERE app IN ('contacts', 'callcampaigns');

See your migration? Marked applied, but table absent? Sync fail.

  1. List tables: \dt in psql. Confirm contacts_callcampaign exists, but no contacts_callcampaign_selected_contacts.

  2. Test schema:

sql
\d contacts_callcampaign_selected_contacts

“Did not find any relation”—confirms.

  1. Docker logs: docker compose logs web db. Spot migrate warnings? Connection timeouts?

  2. Migration files present? In container: ls apps/callcampaigns/migrations/ (whatever your app). Pulled from Git? makemigrations skips if Django thinks current.

  3. App order: INSTALLED_APPS—contacts before callcampaigns? Reverse it temporarily.

Pythontutorials guide lists these as top causes: out-of-sync files, silent fails. Yours ticks multiple.

Quick win? Run showmigrations—arrows show applied status.

Immediate Fixes for Your Setup

No data loss needed. Target the inconsistency.

Option 1: Fake the migration reverse + reapply (safest first)
Exec into web container:

bash
python manage.py migrate callcampaigns 0001 --fake # Whatever your migration number
python manage.py migrate callcampaigns

Django replays just the CREATE, building the table. SO on Postgres tables swears by this for schema mismatches.

Option 2: Manual django_migrations cleanup
In psql:

sql
DELETE FROM django_migrations WHERE app='callcampaigns' AND name='0001_initial'; -- Your file

Then migrate. Multiple SO threads fix ghosts this way. Risky if multi-step, but your CreateModel is atomic.

Option 3: Manual table create (quick hack)

sql
CREATE TABLE contacts_callcampaign_selected_contacts (
 id BIGSERIAL PRIMARY KEY,
 callcampaign_id BIGINT REFERENCES contacts_callcampaign(id) DEFERRABLE INITIALLY DEFERRED,
 contact_id BIGINT REFERENCES contacts_contact(id) DEFERRABLE INITIALLY DEFERRED,
 UNIQUE(callcampaign_id, contact_id)
);
CREATE INDEX ... -- Match Django's auto-index

Then fake-initial. Dirty, but works per Postgres tips.

Test: c.selected_contacts.all() post-fix.

Update Docker command: Add python manage.py showmigrations before migrate for visibility.

Advanced Troubleshooting and Resets

If basics flop?

Full migration reset (data-safe if backed up)

  1. Dump DB: pg_dump.
  2. Delete migrations dir (keep __init__.py), rm apps/*/migrations/0*.py.
  3. python manage.py makemigrations.
  4. Wipe django_migrations for apps: DELETE FROM django_migrations WHERE app IN ('contacts', 'callcampaigns');.
  5. migrate.

SO clean slate and Django Forum endorse for Postgres Docker woes.

Docker tweak: Ensure COPY . /app grabs migrations. Commit them to Git—.gitignore off!

Git issues? This thread blames poor Git hygiene.

Still stuck? Check schema: contacts.Contact table exists with PK? No orphaned FKs.

Nuke volume only as last resort—loses data.

Best Practices for ManyToManyField Migrations in Docker

Commit migrations to Git. Always. .gitignore them? You’re begging for this.

Dockerfile/entrypoint:

yaml
command: >
 sh -c "
 python manage.py wait-for-db &&
 python manage.py migrate --fake-initial &&
 python manage.py collectstatic --noinput &&
 gunicorn ...
 "

--fake-initial skips if tables exist.

Local mirror: Run Docker locally with same volume mount. Test migrations there.

App deps: Explicit dependencies=[] in migrations if cross-app ManyToManyField.

Postgres: Set CONSTRAINTS=IMMUTABLE or tweak DEFERRABLE in through-model if custom.

Better Simple blog calls migrations “difficult”—embrace CI/CD with migrate tests.

Multi-stage Docker: Bake migrations into image.

Preventing Future Issues

  1. Git all migrations: echo "*.pyc" >> .gitignore, but /migrations/ no.
  2. Docker health: wait-for-it script before migrate.
  3. CI pipeline: GitHub Actions runs makemigrations --check --dry-run.
  4. Backup ritual: Cron pg_dump to S3.
  5. Showmigrations always: Log it, alert on inconsistencies.
  6. Separate migrate service: docker compose run --rm migrate python manage.py migrate.

Reddit Docker migrations stresses container inclusion.

Your persistent volume shines here—data safe, just sync the schema.

Wheel back to runtime: smooth selected_contacts.all(). Crisis averted. What’s your next Django adventure?

Authors
NeuroAnswers
Author
Verified by moderation
NeuroAnswers
Moderation
Django ManyToManyField Missing Table: Docker Postgres Fix