Deployment

Web app deployment

The web app is deployed to cabotage automatically whenever a PR is merged to main. Open pull requests are not deployed; see Pull request deployments for why.

Release step and migrations

The release line in the Procfile runs before the new version starts serving. It creates the cache table and runs python manage.py migrate against the production database. Every migration file that reaches main is therefore applied to production on the next deploy, without anyone confirming it.

That gives migrations one rule: a migration is frozen once it is merged.

• While a pull request is open, its migration files may be regenerated, renamed or squashed freely. Nothing applies them except a developer's own database and the CI test run.
• After the pull request is merged, the migration is part of production's history. Never edit it in place, delete it or renumber it. Any further schema change is a new migration file, even a one-line fix.
• If production ever diverges from the migration files (a column missing, a migration recorded as applied that did something else), repair it with a new migration that checks before it acts, such as a RunSQL with ADD COLUMN IF NOT EXISTS. speakers/migrations/0002_repair_invitation_sent_to.py is an example.

Keep pull requests to one migration per app: squash the steps a branch accumulated into a single regenerated file before asking for review, and check it with makemigrations --check and a migrate on a fresh database.

Pull request deployments

Cabotage can build and deploy every push of an open pull request as a preview environment. This was disabled on 2026-09-19 and should stay disabled.

The preview environments shared the production database. Each push of a pull request ran the same release step as a real deploy, so migrate was applied to production from code that had not been reviewed or merged, and that might never merge at all. Two things went wrong with that:

• A migration that was later changed on the branch (regenerated, squashed into another file, or dropped) had already been recorded as applied in production. Django then skipped the merged version, and production ended up with a schema that matched neither the branch nor main. This is how the speakers_invitation.sent_to column went missing and every session page returned a 500 until #414 added it back.
• A pull request that is abandoned or rejected still leaves its tables and columns behind in production, with nothing in main describing them.

With pull request deployments off, only main touches production. Pull requests are verified by the CI test run, which builds its own database from the migration files, and by the Netlify preview for documentation. If preview environments are wanted again, they need a database of their own, separate from production, before they are re-enabled.

Processes

Each line in the Procfile is a process that cabotage runs as its own deployment: web serves requests, worker runs Celery tasks, and worker-beat runs the Celery beat scheduler that dispatches the periodic tasks stored in the database. release runs once per deploy, before the others start.

Process names have to avoid cabotage's reserved names. beat is one of them, which is why the scheduler is called worker-beat.

A new or renamed process starts at zero replicas. Cabotage does not scale it automatically, and a deploy that adds or renames a Procfile entry reports success without ever running the new process. After the deploy, open the application in cabotage and scale the new process to the intended count, typically 1. A renamed process is a new process from cabotage's point of view: scale the new name up and the old name down.

worker-beat must run as exactly one replica. Two schedulers would dispatch every periodic task twice.

If a scheduled task never fires, check the replica count before anything else. In the Django admin, under Periodic tasks, a task whose Last run at is empty has never been dispatched, which almost always means worker-beat is not running.

Emails queued but never sent

An email is queued by the web process and sent by the worker process. The presenter page calls an invitation Sent the moment it is queued, and the record that the email actually went out (a row under Maintenance > Emails) is written by the worker. If the worker is killed or restarted while it holds the task, the task is lost without an error: nothing in Sentry, no failed record, and the page still says Sent. This happened on 1 October 2026.

Maintenance > Invitations (maintainers only) finds these. It lists the active edition's invitations that were marked sent and have no successful email record for that send, oldest first:

Email record column	Meaning
No record	Nothing was recorded. The task was most likely lost.
Failed	The worker tried and the mail provider refused; the error is shown.
Probably still queued	Marked sent in the last five minutes. Wait; its button is disabled for now.

Check a row, then press its Send again button. It is one invitation at a time on purpose: the email goes to a real person and the link in any earlier email stops working, since each send gets a fresh link and expiry. Press it once; a second press within five minutes is refused. A row leaves the list once its email is recorded. If it is still there a few minutes later, the send is failing again: look in the worker log (below). Every send from this page is an activity entry on the presenter and the session ("Sent again from Maintenance") and appears at the bottom of the page.

Limits worth knowing:

• Only invitations that are still waiting are listed. One that was opened, accepted, declined or cancelled is not missing anything.
• Only sends made since the portal started recording emails can be checked. The page says from when: it is when the migration that created the record table was applied, not the age of the oldest surviving record, so a lost first send is still found and the nightly prune cannot hide one. If the worker kept running older code for a while after that deploy, its sends in that window have no record though they were delivered, and are listed.
• If the mail provider accepted a message and the process died before the record was saved, the speaker has the email but the page lists it. Sending again gives them a second one.

Check the workers after every deploy. Every worker reading the default queue takes tasks, and a task goes to whichever takes it first. A release can leave an older worker pod running beside the new one (this happened on 1 October 2026, and both showed under the one process), and then some emails are sent with the old code: without a record, without the log lines, with the old behaviour, while the deploy reports success. The top of Maintenance > Invitations lists every worker attached to the broker: its name, which queue it reads, how long it has been up, and whether it runs the same email code as the site. The check after a deploy is one default worker, saying same as this site, and one media worker (a worker started with -Q celery,media, as in development, counts as both). Red alerts name the problem: more than one worker reading the default queue, none reading the default queue, none reading the media queue (thumbnails, transcription and downloads wait), or a worker whose code the page cannot confirm. That last one reads one of four ways: not reported means the worker answered that it does not know the command, so it runs code older than the page and should be removed; differs shows a different fingerprint of the files that decide what an email task does (common.workers.email_code_version); error is a worker that knows the command and failed, with its error; no answer is a worker that was attached but did not reply in time, which may be busy or stuck. The fingerprint is of the code the process started with, so a changed file shows the old value until the process restarts.

The page takes about a second longer to load while it waits for answers. If the broker cannot be reached it says so within a few seconds instead of hanging: connecting is bounded by CELERY_BROKER_TRANSPORT_OPTIONS (socket_connect_timeout, which also bounds a worker's own connect), and the page gives up after the wait plus a few seconds. The same list is available from a shell with celery -A portal inspect ping.

Finding out why. Cabotage runs each Procfile line as its own deployment, so the worker process has its own logs, apart from web; open that process, and check it is scaled to at least one replica. The apps log at INFO to standard output (PORTAL_LOG_LEVEL changes the level). For one invitation the lines are, in order:

INFO speakers.services: Invitation 482 for presenter 91 marked sent by 7; email queued on commit
INFO common.tasks: Queued speakers.tasks.send_invitation_email_task as task 3f1c...
INFO speakers.tasks: Invitation email: sending invitation 482 to presenter 91 (task 3f1c...)
INFO speakers.tasks: Invitation email: invitation 482 sent, email record 1042 (task 3f1c...)

The task id ties the web log to the worker log; Celery's own received and succeeded lines carry the same id. A received line with no succeeded, or a WorkerLostError or "Worker exited prematurely" line, means the worker died holding the task: look for restarts and out-of-memory kills of the worker process at that time. No received line at all means the task never arrived: search the web log for Failed to enqueue Celery task, which is a broker (Redis) outage. A failure inside the task logs the traceback and the invitation id, and goes to Sentry. Addresses are never logged.

Celery starts one child process per CPU it can see, which in a container is often the node's core count, not the container's share; a small memory limit then gets a child killed. Set --concurrency on the worker line of the Procfile if restarts coincide with lost emails. The design for tracking delivery properly, and the steps not built yet, are in docs/architecture/email-delivery.md.

Database

The portal needs PostgreSQL 15 or newer. The speakers app declares unique constraints with nulls_distinct=False (so a row with a NULL presenter still counts as a duplicate of another). Django only emits that clause on Postgres 15+; on an older server it warns with models.W047 at check time and creates the constraint without it, which silently allows the duplicates the constraint exists to prevent. Production runs PostgreSQL 17 (17.2 as of September 2026); local development and CI run postgres:16 from compose.yml, one major version behind, which is fine for the features the portal uses. The image ships no psql, so manage.py dbshell does not work in production; ask Django instead:

python manage.py shell -c "from django.db import connection; print(connection.pg_version)"

The number reads as major and minor: 170002 is 17.2.

Adding a column during a rolling deploy

The release step runs migrate and only then serves the new release, so there is a window, usually seconds, where the schema is new and the code is old. A column added as NOT NULL breaks that window: Django backfills the existing rows with a default and then drops it, so the column has no database default, and a process from the previous release inserts a row without it. Postgres refuses:

null value in column "default_team_name" of relation
"speakers_checklisttemplateitem" violates not-null constraint

It clears up on its own once the new release is serving, but it is 500s for real people in the meantime, and it is avoidable. When adding a NOT NULL column, give it a database default as well as a Python one:

default_team_name = models.CharField(max_length=40, blank=True, db_default="")

db_default (Django 5) keeps the default in the schema, so the old code can keep inserting until it is replaced. Keep the ordinary default as well: with only a db_default, an unsaved instance holds a sentinel rather than the value, and anything reading the attribute before the row is saved, a clean() for instance, sees the sentinel.

tests/speakers/test_deploy_window.py walks the migration graph for every column added to a table that already existed and fails if one is NOT NULL with nothing behind it, so the rule cannot drift as columns are added. A foreign key is the case where no default exists: add it nullable, backfill it, and tighten it in a later release.

The same window applies to a column that is later removed: drop it from the model first, deploy, then delete the column in a follow-up migration.

Speaker media bucket

Speaker media (design §8.8) lives in its own private bucket, named by SPEAKER_MEDIA_BUCKET, never the public-read one the image fields use. It needs a lifecycle rule that aborts incomplete multipart uploads after 7 days (AbortIncompleteMultipartUpload): the portal's nightly "Expire abandoned uploads" task reclaims what it knows about, and the rule reclaims the rest. The nightly "Expire export zips" task deletes the zip of every bulk-download export past its expiry. The bucket also needs a CORS rule allowing PUT from the portal's origin with the ETag header exposed, since the browser uploads the parts directly. Bulk download (design §8.8, task 5.6) needs GET in the same rule for the browser's folder download, and a second lifecycle rule that expires the exports/ prefix after 7 days, since zip bundles are built there and their links die with the export after twelve hours.

Endpoint and region. The media client reads SPEAKER_MEDIA_ENDPOINT_URL and SPEAKER_MEDIA_REGION, each falling back to AWS_S3_ENDPOINT_URL and AWS_S3_REGION_NAME when unset or blank. Set them to the media Space's regional host and region (https://sfo3.digitaloceanspaces.com and sfo3, say), not a Space's own host: boto3 puts the bucket name in the path, so a host that already names another Space signs every upload for the wrong Space, and the browser's preflight fails with no Access-Control-Allow-Origin. A region that is not the Space's gives a signature error. The image storage keeps using the AWS_* variables unchanged. The access key must reach both Spaces.

Read-only key for a post-production lead. The bulk-download design names one escape hatch that needs no portal code: a Spaces key with read-only access to the media bucket, so one trusted person can pull an edition with rclone. Create it under API, Spaces Keys, scoped to the media bucket only with Read permission, and hand it over through the password manager, never by email or chat. Revoke it when the edition's post-production is done; nothing in the portal records what such a key fetches.

rclone config create do-media s3 provider=DigitalOcean \
    endpoint=nyc3.digitaloceanspaces.com \
    access_key_id=<key> secret_access_key=<secret>
rclone sync do-media:pyladiescon-media/speaker-media/2026 ./pyladiescon-2026 \
    --progress --transfers 4

The media worker

Media jobs run for minutes at a time: the duration probe (task 5.3, built) measures every video that lands with ffprobe, and machine transcription (task 5.7, designed) will run Whisper. They go on the media Celery queue (CELERY_TASK_ROUTES in settings), served by the worker-media process in the Procfile rather than the default worker, so a half-hour transcription never holds up an invitation email.

• The image installs ffmpeg, which brings ffprobe.
• worker-media is a new process, so it starts at zero replicas in cabotage and has to be scaled to one by hand after the deploy that adds it (see "A new or renamed process" above); keep it at one replica, and it runs with concurrency 1. Until it is scaled up, every uploaded video shows "Length not checked yet" for good: the probe is queued and nobody is consuming the queue.
• A probe that fails records why on the asset ("Length not measured: ffprobe is not installed on the worker", for instance), which the file rows show, and logs an error; the asset itself stays usable.

Transcription (task 5.7, built):

• the library and the Whisper model are part of the image by default: the Dockerfile's ARG WHISPER_MODEL defaults to small, so a platform that cannot pass build arguments (cabotage) still gets an engine. The layer installs requirements-media.txt and downloads the model into /opt/whisper before the code is copied, so a code-only deploy reuses the cached layer. compose.yml sets the argument to empty for the dev images (and so for CI), which carry neither; build with --build-arg WHISPER_MODEL= to get a lean production image, and an image without a model has no engine, which the job says on the page. The worker never downloads at run time; changing the model is an image rebuild; give the process 2 GB of memory for small, more for medium;
• set SPEAKER_TRANSCRIBE_ENGINE=local (and SPEAKER_TRANSCRIBE_MODEL to the model that was baked in) on every process, since the web process reads the first to offer "Transcribe this"; then switch Auto transcribe on per edition under Speaker settings for automatic drafts;
• transcription runs entirely in this process; no audio leaves the portal's infrastructure, and there is no API key to manage.

What a second worker process costs. It is one more container on the PSF-hosted cabotage cluster with its own memory reservation (the Whisper model is what sets it), one more line to scale by hand after the deploy, and one more thing that can be at zero replicas without anything failing loudly: media tasks sit in Redis until a worker-media picks them up, which is why the nightly watchdog marks a job queued for over twelve hours as failed on the page. It shares the image, so ffmpeg and the faster-whisper dependency are in every process's image whether or not they use them (about 250 MB more image, plus about 500 MB for the Whisper model layer). Everything else is unchanged: web, worker-beat and release do not know it exists, and the default worker keeps consuming only its own queue.

The alternative is one worker consuming both queues with two child processes. It saves the container and the manual scale-up, but the whole worker then needs the model's memory, a transcription and an email share the same limits, and two slow transcriptions would hold every email until one finishes. The separate process is the recommended shape; the single worker is the fallback if the cluster cannot spare a container, and it is a Procfile and settings change, not a code change.

Email and worker settings

These are read from the environment of every process. All have defaults, so none is required.

Variable	Default	Meaning
DJANGO_EMAIL_TIMEOUT	30	Seconds a connection to the mail server may wait. Applies when DJANGO_EMAIL_HOST is set. Without a timeout, a server that accepts the connection and then says nothing holds a worker process for ever.
DJANGO_EMAIL_HOST_PASSWORD	none	The SMTP password. The old, misspelt name DJANOG_EMAIL_HOST_PASSWORD is still read, so an existing deployment keeps working; the correctly spelt one wins when both are set.
CELERY_WORKER_CONCURRENCY	2	Processes per worker. Celery's own default is one per CPU it can see, which in a container is often the node's core count rather than the container's share; a small memory limit then gets a process killed, and the task it held with it. worker-media sets its own on the command line.
PORTAL_LOG_LEVEL	INFO	Level for the apps' own log lines.

Workers hand each process one task at a time (CELERY_WORKER_PREFETCH_MULTIPLIER is 1), so a long task does not leave others reserved behind it while another process sits idle. The broker connection is bounded and kept alive (CELERY_BROKER_TRANSPORT_OPTIONS).

What the email tasks do when something goes wrong. The tasks that send one email to one person (the invitation, the acceptance and added-to-session emails, and the approved and rejected proposal replies) are acknowledged when they finish, not when they start, so a worker that is killed or restarted while holding one gives it back to the queue. They retry a mail server that is down or slow (four times, with growing waits), and give up at once on an address the server refuses or a login it rejects. Each has a time limit, so a stalled connection cannot hold a process. A task that runs twice (a worker that dies after the mail server accepted the message) checks first whether the send is already on record and does not mail the person again. A worker logs Worker ready: <name> process=<kind> email_code=<fingerprint> each time it starts, so a worker that restarts every few minutes shows as a run of those lines, and each one says which code it came up with. Sentry events carry a process tag (web, worker, worker-media or beat) so an error can be traced to the kind of process it came from. The digests and the notices to the organizers are not retried, because a retry would send again to people who already have them.

One-time configuration

Two things live outside the code and have to be set on a new environment:

Site domain. Emailed links (account verification, speaker invitations, checklist reminders) are built from the domain in Django's sites framework, which starts as example.com. After the first deploy, set it to the portal's public hostname, either from the release shell:

python manage.py set_site_domain portal.example.org

or in the Django admin under Sites. Links are https:// whenever DEBUG is off.

FERNET_KEYS environment variable. The speaker portal stores the pretix API token and webhook secret encrypted at rest and refuses to save them without a key. Generate one once and keep it with the other deployment secrets:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

The variable is comma-separated so keys can be rotated: the first key encrypts, every key decrypts. To roll a key, put the new one first, deploy, re-save the stored secrets from the admin, then drop the old key. A single FERNET_KEY is accepted too. A value stored under a key that is no longer configured reads as unusable (pretix shows as not configured, and the app logs it once) rather than breaking every page that loads the settings; re-entering the secrets in the admin repairs it. Local development and the test suite derive a key from SECRET_KEY, so neither needs the variable.

Documentation deployment

The documentation is deployed to Netlify automatically whenever the PR is merged.

There is also a preview of the documentation for each PR, also generated by Netlify.