Backing up and restoring a self-hosted EntryWick (week 21)

What is backed up on a self-hosted server, how to take and schedule backups with entrywick backup, how to keep a copy off the server, and how to restore — on the same server or a new one. ADR-019 (backups are the operator's), docs/runbooks/restore.md (the week 19 backups this builds on). The commands' reference: cli.md.

Who opens it: whoever runs a self-hosted install — once to set backups up, and on the bad day they need one.

What they came to do: make sure a lost server or a bad change costs at most a day, and put a backup back.

What must be true when they leave: a complete backup is taken every night, a copy of it lives off the server, the .env that reads it is in their password manager, and they know what a restore replaces and what it loses.

Two backups run on a self-hosted server

Both are safe during an event: the dump is taken in one transaction and nothing is stopped.

What a backup holds

entrywick backup writes a directory named for the time it started (UTC), such as backups/20270215-041500/, mode 700:

Not in it: Redis (queued jobs in flight are lost and scheduled work runs again on its next tick; live counters are rebuilt), Caddy's certificates (issued again on the first visit), and .env.version (the restore keeps the release that runs).

A backup is as secret as .env: anyone with it can read every attendee's data. Keep copies of it as carefully.

Taking one

entrywick backup

It ends Backup 20270215-041500 in /opt/entrywick/backups/20270215-041500. It holds a copy of .env: keep it as safe as .env itself. The oldest backups past five are deleted (pruned backup …); --keep=N keeps another number.

entrywick backup --list

lists them newest first: name, time, the release that made it, and its size in MB.

entrywick backup refuses when EW_BACKUP_DISK in .env is anything but local (the installer writes local).

Every night

Schedule entrywick backup with cron, after the app's own nightly jobs (02:45–03:45 UTC):

cat > /etc/cron.d/entrywick-backup <<'EOF'
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
15 4 * * * root /usr/local/bin/entrywick backup >> /var/log/entrywick-backup.log 2>&1
EOF

That runs at 04:15 in the server's timezone (UTC on most VPS images: check with timedatectl). The next morning, entrywick backup --list shows it, and /var/log/entrywick-backup.log says why when it does not. entrywick doctor warns when the newest backup on the server is more than 7 days old.

Five nightly backups are five days. Each one holds every stored file again, so watch the disk (requirements.md); entrywick doctor warns under 10 GB free.

A copy off the server

A backup on the server is lost with the server. Keep a copy somewhere else, in one or both of these ways.

The offsite service (nightly, to a bucket)

The same service EntryWick's own servers use (docs/runbooks/restore.md). Every night at 03:30 UTC it copies the storage volume to an S3-compatible bucket — which holds the app's nightly database dumps and every stored file. It does not copy /opt/entrywick/backups or .env: keep .env in your password manager.

1. A bucket: a private bucket at Backblaze B2 or any S3-compatible store, with encryption on and a rule that keeps prior versions for 30 days, and a key that can read and write only that bucket (docs/runbooks/restore.md, "Setting it up", step 1).
2. Add to /opt/entrywick/.env, for B2:

   EW_BACKUP_REMOTE=offsite:your-bucket/entrywick
   RCLONE_CONFIG_OFFSITE_TYPE=b2
   RCLONE_CONFIG_OFFSITE_ACCOUNT=<keyID>
   RCLONE_CONFIG_OFFSITE_KEY=<applicationKey>
   

   For S3 and others: RCLONE_CONFIG_OFFSITE_TYPE=s3 with …_PROVIDER, …_ENDPOINT, …_ACCESS_KEY_ID, …_SECRET_ACCESS_KEY and …_REGION.
3. Start it, and make the first copy now:

   entrywick compose --profile backups up -d offsite
   entrywick compose --profile backups run --rm offsite once
   

4. Check: entrywick doctor shows offsite copy: 0 h ago. It says late when a copy is more than 24 hours old.

Copying the backup directories

entrywick restore needs a whole backup directory. To keep them elsewhere too, copy /opt/entrywick/backups/ from another machine you trust after the nightly backup, for example:

rsync -a root@203.0.113.10:/opt/entrywick/backups/ ./entrywick-backups/

Restoring on the same server

entrywick restore 20270215-041500

Before it touches anything, it:

• checks every file against its checksum (backup …: every file matches its checksum), and stops if one does not;
• refuses a backup made by a newer release than the one running (update first);
• refuses while an event is live, unless --force (a restore loses the check-ins made since the backup);
• asks you to type the backup's name: This replaces the database and stored files with backup 20270215-041500 (…); everything since is lost. Type the backup's name to go on:. Anything else ends with Nothing was restored. (exit 1). --force skips this question too.

Then it stops Horizon and the scheduler, unpacks the stored files over the volume (files added since the backup stay), loads the database, migrates it up to the running release, re-seeds EntryWick's own rows, rebuilds the live counters, restarts everything and checks it answers. It ends Restored backup 20270215-041500 (taken …). What was written after it — registrations, check-ins, changes — is not in it. Paired scanners keep working.

APP_KEY: when the backup's APP_KEY differs from the one in .env, it replaces it (warn APP_KEY in .env now is the backup's …), because the restored database was encrypted with it. Everyone signs in again. Every other line of .env stays as it is.

Restoring onto a new server

From a backup directory

1. Install EntryWick on the new server (install.md) with the same domain and license key, and the same release or a newer one. Point the DNS records at the new server. The organization and Owner the install creates are replaced by the backup's.
2. Copy the backup directory into /opt/entrywick/backups/ on the new server, for example from the machine that holds it:

   ssh root@NEW-SERVER mkdir -p -m 700 /opt/entrywick/backups
   rsync -a ./entrywick-backups/20270215-041500 root@NEW-SERVER:/opt/entrywick/backups/
   

3. On the new server:

   entrywick restore 20270215-041500
   

4. Sign in with the accounts from the backup, and run entrywick doctor. Scanners paired before keep working. Set up the nightly backup and the offsite copy again.

From the offsite copy only

When the server and its backups/ directory are gone, the bucket still has the nightly dumps and the stored files. You need the old .env from your password manager.

1. Install EntryWick on the new server (install.md) with the same domain and license key.
2. In /opt/entrywick/.env, replace the APP_KEY= line with the old one, and add the four offsite lines from "The offsite service" above. Leave the database passwords as the install wrote them.
3. Bring the files and dumps back, and load the newest dump:

   entrywick compose stop horizon scheduler
   entrywick compose --profile backups run --rm offsite pull
   entrywick compose up -d --wait --force-recreate app
   entrywick compose exec app php artisan ew:restore --force --i-know-this-replaces-production
   entrywick compose exec app php artisan migrate --force
   entrywick compose exec app php artisan db:seed --class='Database\Seeders\PlatformSeeder' --force
   entrywick compose exec app php artisan ew:rebuild-counters
   entrywick compose up -d --force-recreate
   entrywick compose --profile backups up -d offsite
   

   ew:restore checks the newest dump against its recorded checksum before loading it.
4. entrywick doctor, then sign in with the old accounts. What was written after that night's dump is not there.