Operations: keep your cluster running#

All guides · Browser management guide

Use this guide for backups, upgrades, maintenance, and recovery. If you only need to deploy, scale, or inspect an application, start with everyday browser management.

A simple daily routine#

Reading a notification does not acknowledge an alert. Acknowledging an alert does not fix it.

Find the right procedure#

• Backups and restore
• Upgrades
• Worker maintenance
• Failure behavior
• Monitoring the control plane
• Troubleshooting

Backups and restore#

There are two separate backup jobs: Zela's management information, and your applications' own data. A Zela backup does not contain your workers' Docker volumes, database files, or application uploads. Arrange backups for those with the application owner.

For Zela, preserve the following from the same installation:

The original vault key is required to decrypt saved SSH keys. An administrator token can be rotated, but preserving the existing credential simplifies recovery. Store sensitive backups privately and keep a protected copy of the vault key separate from the database backup.

With JSON storage, stop the manager and back up the whole --data directory, including its keys and tokens. Individual JSON files are replaced atomically, but copying several live files does not create a coordinated backup of the whole installation.

Example: back up the supplied Compose installation#

Run these on the manager computer, in the project folder. The backup folder contains secrets; keep it out of source control. Use a new backup folder for each backup; this example uses zela-backup-1. No cluster operation should be running. Stopping the manager temporarily disconnects the dashboard; existing worker containers continue running.

mkdir -m 700 zela-backup-1
docker compose stop zela-server
docker compose exec -T postgres pg_dump -U zela -d zela -Fc > zela-backup-1/zela.dump
docker compose cp zela-server:/data zela-backup-1/data
cp .env zela-backup-1/compose.env
chmod -R go-rwx zela-backup-1
docker compose start zela-server

Check each command before continuing. If a backup command fails, resolve it and restart the manager as needed; a folder existing does not mean the backup succeeded. Confirm the dump is readable with your database backup tools, preserve the copied vault key, and transfer the backup securely off this computer. If your configuration supplies tokens or keys outside /data, preserve those separately too. Test restoration on an isolated installation before relying on the backup.

For other installations, have the database administrator coordinate a PostgreSQL backup and protected copies of the data directory and externally supplied secrets.

Restore#

1. Stop every manager.
2. Restore the schema: pg_restore --clean -d zela zela.dump.
3. Provide the original vault key and admin token.
4. Start a manager. Operations that were running at backup time show as failed; inspect them before retrying.

Restored state may be older than reality. After the first inventory poll, compare Containers with Deployments and remove any leftovers.

Upgrades#

Plan a short maintenance window. Wait for current operations to finish, back up first, and keep your original credentials and data volumes. Updating files does not update a manager that is already running.

For the supplied Compose installation, from the project folder containing the updated source and your original .env:

docker compose up --build -d zela-server
docker compose ps
docker compose logs --tail=80 zela-server

Check that sign-in works, workers are online, and a sample application still responds. Update worker agents afterward as described below. Do not delete volumes or create a new database password to perform an upgrade.

For a binary/service installation:

1. Read the release notes for schema migrations and agent protocol changes.
2. Build or download the new release (make linux).
3. Back up (see above).
4. Upgrade managers. Replace the zela-server binary and the worker artifacts in --worker-binaries, then restart. Migrations run automatically on start.
   • HA: upgrade all managers in the same maintenance window. A manager that sees a newer schema version refuses to start, so don't roll back an individual manager after the database has been migrated. Restore the backup if you need to downgrade.
   • Schema version 2 adds the HA leadership-epoch table. Managers built before it refuse to start against a migrated database.
5. Upgrade agents.
   • SSH-provisioned workers: Monitoring → Update agent, or zela workers update-agent <id> --wait. This preserves the token, service flags, placement mode and running containers.
   • Manual agents: copy the new zela-worker and run systemctl restart worker.
6. Upgrade zela on admin machines.

Restarting an agent doesn't affect running containers. Restarting a manager interrupts its running operation, which is marked failed, and drops open Docker streams. Clients must reconnect.

Worker maintenance#

Drain moves only managed, stateless replicas. Containers with mounts, stopped deployments, missing capacity, or unmanaged containers leave the worker in draining until you resolve them. Unregistering never uninstalls Docker or the agent.

Failure behavior#

A worker goes down#

Recovery is not possible when no other worker matches the deployment's placement (WorkerIDs) or has capacity. The deployment then stays degraded with an error. Fix capacity or placement, then run zela deployments start <id> to retry.

Containers created through the Docker gateway (Docker CLI and SDK clients), and containers that weren't created by Zela, are not recreated automatically. Their owner handles recovery. An offline owner returns 503 for their requests.

The database becomes unavailable#

• Standalone: the manager checks persistence every 10 seconds and exits when the session is lost, so systemd or Compose restarts it. A session never reconnects silently. Workloads keep running on workers.
• HA: the leader stops reporting ready and fences itself if checks keep failing for about 5 seconds. Standbys keep trying to take the lock. See HA failover.

A manager crashes#

The interrupted operation is marked failed on the next start (or by the new leader in HA). It is not replayed, because a Docker mutation may already have happened. Inspect it in Activity before retrying.

Monitoring the control plane#

Troubleshooting#

Start with Activity log for a failed change, the worker's details for a connection error, and container logs for an app that will not start. Wait for a running operation before trying another change. If a request times out, inspect the current state before repeating it.

The table below is for the person investigating the server or connection configuration.

For more detail, run the manager in the foreground (./bin/zela-server --data ./data ...) and reproduce the problem. Review log output for application secrets before sharing it; application logs and error text can contain sensitive information.