Server operations

Daemons & cron

Queue workers, long-running processes, and scheduled commands, without writing unit files.

Daemons

A daemon is a long-running command that systemd supervises and restarts when it exits. Add one under Daemons on a server's Processes page. Give it a command, a working directory, and between 1 and 10 processes. Start, stop, and restart it from its row. Changing the command re-applies the unit.

A single-process daemon is a plain unit, provisore-daemon-<name>.service. Ask for more than one process and it becomes a systemd template unit, provisore-daemon-<name>@.service, run as provisore-daemon-<name>@1 through @N. Provisore reads all instances together, so a multi-process daemon's log shows every instance's output.

A deploy restarts the site's daemons on the release it just made live, before the health check. If they cannot start on it, the deploy fails and rolls back. A worker never keeps running the previous release's code behind a new release.

A Laravel queue worker

Command    php artisan queue:work --sleep=3 --tries=3 --max-time=3600
Directory  /home/provisore/sites/example.com/current
Processes  2
Site       example.com

Cron entries

Add scheduled commands under Scheduled tasks on the same Processes page. Provisore writes each one to /etc/cron.d/provisore-<name> rather than a user's crontab, so it can update or remove exactly what it installed. The schedule is a standard five-field cron expression, validated before anything is written.

An entry that names a site runs in that site's current release, with the site's pinned PHP and Node ahead of everything else on PATH. Write the command on its own, with no cd and no absolute path to an interpreter.

The Laravel scheduler

Schedule   * * * * *
Command    php artisan schedule:run
Site       example.com

A Laravel site already has this one

Creating a Laravel site creates its scheduler entry. Removing the entry stops every Schedule:: line in your repository, with no error and no failed job. Delete it only if the site schedules nothing.

Tying them to a site

A daemon or cron entry can name one of the team's sites on the same server. The entry then runs as the site's own Unix account, the only account that can read the site's shared/.env. A queue worker running as the deploy user cannot read its application's environment.

An entry with no site runs as provisore or www-data, whichever you choose.

Publishing a daemon at a domain

To serve a daemon's port at a hostname, create a web service on the same server, give it the port, and name the daemon as the process behind it. Nginx terminates TLS for the domain and forwards to that port.

Naming the daemon records what listens on the port. It does not transfer ownership. Several web services can name one daemon, and while any of them do, the API refuses to delete it with 409 daemon_in_use and lists the domains. Unbind it from each, or delete those web services, and the delete goes through.

When a daemon restarts forever

Every unit Provisore writes carries Restart=always and RestartSec=3. That suits a queue worker and punishes a command that cannot start at all. If the binary the command names is not on the server, systemd tries to start it, fails, waits three seconds, and tries again for as long as the server is up.

The daemon's row will not show it. Creating a daemon runs systemctl restart, which succeeds as soon as systemd accepts the unit, not once the process has stayed up. The job succeeds and the daemon reads as running. Open the daemon's log from its row and the journal shows what happened:

A command that is not on the server

systemd[1]: provisore-daemon-relay.service: Main process exited, code=exited, status=203/EXEC
systemd[1]: provisore-daemon-relay.service: Failed with result 'exit-code'.
systemd[1]: provisore-daemon-relay.service: Scheduled restart job, restart counter is at 102.

203/EXEC means systemd could not execute the command. Put the binary on the server first, then create the daemon. Two common causes:

A daemon that names a site gets that site's shims first on PATH, so its pinned PHP and Node resolve by name. A server-wide daemon gets systemd's default PATH, /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, and nothing else. A command that resolves in your SSH session through your shell profile may not resolve there. Use an absolute path when unsure.
A command that runs from a release directory points at a path that exists only after the site has deployed. Deploy the site first.

When a daemon cannot find a library

A binary you built elsewhere and copied to the server links against shared libraries, and the server has only what its own packages installed. When one is missing, the dynamic loader stops the process before your code runs and systemd records exit status 127:

A library the server does not have

relay: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory
systemd[1]: provisore-daemon-relay.service: Main process exited, code=exited, status=127

Provisore installs the toolchain a site's runtime needs to build and run that site. It does not manage the libraries of a binary you brought yourself. Ask the binary what it is missing, over SSH or as a one-off command:

ldd /usr/local/bin/relay | grep 'not found'

apt-file search libpq.so.5 names the package that provides each one. Install them and restart the daemon. Building on the server, or on the Ubuntu release the server runs, avoids the problem.