Skip to content

Troubleshooting servers

Fixes for adding a server, keeping it connected and keeping its disk free. Commands with sudo run on the server; sudo journalctl -u ployz -n 200 shows Ployz's logs there.

You'll see sudo: command not found or run this command with sudo.

  • sudo: command not found: you're root on a server without sudo. The CLI is already installed, so run the part after && again without sudo, and paste the token when it asks:

    ployz server add
  • run this command with sudo: keep the sudo the dashboard command includes, or run it as root.

You'll see Unsupported platform, Ployz requires systemd, unsupported Machine architecture or Ployz Machine must be Linux.

Use an amd64 (x86_64) or arm64 (aarch64) server running a Linux distribution with systemd.

The server can't set up managed volumes as it is. Find your message below for the fix. Either way, you can add --storage none to the end of the command and run it again: the server shows Docker only; see Start without ZFS for what that costs.

  • Managed volumes need Ubuntu LTS, Debian 12–13 or Amazon Linux 2023; this Server runs ... Use one of those, or add `--storage none`.

    The server runs another distribution, or another version of one of these. Reinstall it with a supported image.

  • Secure Boot is on, so this Server can't load the ZFS module Ployz builds for ... Turn Secure Boot off, use Ubuntu, or add `--storage none`.

    Turn Secure Boot off in your provider's settings for the server, or reinstall it with Ubuntu. Then run the command again.

  • Amazon Linux 2023 has no kernel-devel package for the running kernel ..., so ZFS can't be built for it. Update the kernel, reboot, and retry, or add `--storage none`.

    The server runs an older kernel than its package repositories offer. Update it, reboot, then run the command again:

    sudo dnf upgrade -y
    sudo reboot
  • Couldn't build ZFS for kernel ... on ...: the OpenZFS checksum check failed. Add `--storage none` to start without managed volumes.

    The ZFS source the server downloaded wasn't the release Ployz expects, so nothing was installed. Run the command again. If it fails the same way, something between the server and GitHub, like a proxy, changes downloads.

  • Couldn't build ZFS for kernel ... on ...: ... failed. Add `--storage none` to start without managed volumes.

    The build stopped at the step the message names. The output above the message shows why. Fix that and run the command again.

  • Ubuntu has no packaged ZFS module for the running kernel ...; install a supported Ubuntu kernel and retry

    Switch the server to Ubuntu's stock kernel, reboot, and run the command again.

  • ... apt-get is required for ZFS storage preparation, or the same for dnf

    Reinstall the server with its provider's stock image.

  • OpenVZ does not allow this Machine to load the host ZFS kernel module, or the same for unprivileged LXC

    Use a VM or bare metal.

  • Host root has ... bytes available; ZFS validation needs ...

    Free disk space, or give the server a bigger disk. Ployz keeps a quarter of the disk, at least 10 GiB, free for the system, so use a main disk of 30 GiB or more.

The server already has a Docker that Ployz can't use. Setup stops before it changes anything, so fix Docker and run the command again. Or uninstall Docker, and Ployz installs a current one.

  • Docker is installed but not running. Start it (systemctl start docker) and run this again.

    Run sudo systemctl start docker, then run the command again.

  • Docker ... is too old: Ployz needs Docker 29.2 or newer. Upgrade Docker, or uninstall it and run this again so Ployz installs a current one.

    Upgrade Docker from the source you installed it from. A distribution's own Docker package, like Amazon Linux's docker, is often older: remove it with sudo dnf remove docker or sudo apt-get remove docker.io, then run the command again.

    Ployz 0.3.2 and earlier then fail to start the Docker they install: restart Docker: Job for docker.service failed. Run sudo systemctl reset-failed docker.socket docker.service and sudo systemctl restart docker.socket docker.service, then run the command again.

  • Docker Buildx ... is too old: Ployz needs Buildx 0.18 or newer. Upgrade Docker, or uninstall it and run this again so Ployz installs a current one.

    Upgrade the buildx plugin along with Docker, or uninstall Docker.

  • Docker isn't using the containerd image store, which Ployz needs to build and move images between Servers. Enable it in /etc/docker/daemon.json ("features": {"containerd-snapshotter": true}) and restart Docker, or uninstall Docker and run this again.

    Add "features": {"containerd-snapshotter": true} to /etc/docker/daemon.json and run sudo systemctl restart docker. Images you pulled before switching don't carry over, so pull them again.

Once Ployz is installed, this prints what it reads from the server and what would stop it, without changing anything:

sudo ployzd install --probe --json

facts holds the distribution, kernel, virtualization, cgroup version, CPUs, memory, disks and Docker. assessment.blockers lists what stops Ployz, each with a code and a message, and assessment.warnings lists what may misbehave. A root disk under 30 GiB shows as the disk_too_small blocker, but the install doesn't refuse it yet. assessment.zfs is Eligible with the disks managed volumes can use, or Ineligible with the reason the ZFS install stops on. A value Ployz couldn't read shows as null, and facts.unread says why. When Ployz can't read whether an LXC container is privileged, assessment.zfs is Ineligible with lxc_privilege_unknown, and when it can't read the firmware's Secure Boot setting on Debian or Amazon Linux, Ineligible with secure_boot_unknown. When a Docker command fails partway, the docker_unreadable blocker carries the error the install would stop on.

The probe never starts Docker. When docker.socket listens but docker.service isn't running, facts.docker is SocketActivated with no version, and facts.unread says why. That isn't a blocker: the install's first Docker command starts it. If docker.service failed the last time it ran, facts.unread says so, because only the install's start shows whether it fails again. A Docker that doesn't answer shows as docker_stopped.

Managed volumes are missing after a kernel update

Section titled “Managed volumes are missing after a kernel update”

After a Debian or Amazon Linux server reboots into a new kernel, its managed volumes are unavailable and the services that use them don't start. Your volume data is still on the disk.

The server rebuilds ZFS for each kernel it installs, but that build failed, for example because the kernel is newer than ZFS supports. Either:

  • rebuild ZFS for the running kernel, then reboot:

    sudo dkms autoinstall
    sudo reboot

    If it fails, install the running kernel's headers first, then try again:

    # Debian
    sudo apt-get install linux-headers-$(uname -r)
    # Amazon Linux
    sudo dnf install "kernel-devel-uname-r = $(uname -r)"
  • or boot the previous kernel, which stays installed. Choose it in the boot menu from your provider's console.

You'll see Cloud no longer takes this enrollment token.

The enrollment token expired or was mistyped, or the organization's servers were forgotten, which voids every token made before. A token works for 24 hours. Run ployz server add --command on your computer for a new one and run its line again.

You'll see web-1 refused: followed by the server's reason, such as that the command needs Full access. The server answered, but this computer's access doesn't cover the command, so nothing changed on it. When the server suggests a next step, the CLI prints it under the reason. When the reason says the command needs Full access, approve this computer as a device. See Approve a device. When it says only Cloud manages slots, make the change from the dashboard instead. Before Ployz 0.3.4, the CLI said such a server did not answer.

Removing a server says Ployz Cloud can't read an environment

Section titled “Removing a server says Ployz Cloud can't read an environment”

You'll see Ployz Cloud can't read shop/production from the Config Store from ployz server rm, from Remove server, or when you move volumes off a server. Ployz Cloud holds settings for that environment it can't read back, as happens when a self-hosted Cloud is rolled back below the release that wrote them. Ployz can't tell which of that environment's volumes are on the server, so it removes nothing rather than risk losing their only copy. The Remove server list still names every other environment's volumes, and lists that environment's volumes as Unreadable volume. If you rolled Ployz Cloud back, roll it forward (see Self-hosting), then try again. Otherwise, contact support.

You'll see invalid Machine Name "Web_1", with the closest valid name, web-1, below it.

A name must be one lowercase word: letters, digits and hyphens, up to 63 characters. Only a name you pass with --name is refused; pass the suggested name and run the command again.

You'll see This Server is already initialised, followed by advice to make a new token.

The server is already in a cluster, or partway through joining or leaving one. The token that add used is now tied to the server as it is, and --reset gives the server a new identity, so that token won't work after a reset. Add it again with --reset and a new token. Signed in, either run ployz server add --command --reset and run the line it prints on the server, or run ployz server add root@HOST --reset to add it over SSH, which makes its own token. The reset removes every container Ployz runs on that server, so its apps go down, and it can delete its mirrors, so don't use it while a mirror there may be the last copy of a volume.

You'll see This Server is already in a Cluster with other Servers, or something other than Ployz Cloud manages it. From Ployz 0.3.15, ployz server add may stop earlier on the server with this Server is already in a Cluster this enrollment didn't found, or with a new founding claim requires an uninitialized Server. When it also says Ployz Cloud let go of this founding attempt, the token is used up and nothing else is left to reset.

An organization's first server starts a new Cluster, so it can't be a server that already has other servers, or that something other than Ployz Cloud manages. The server stays in its Cluster. Don't run another token on it as it is.

Run ployz server add --command for a new token, and add a server that isn't in a Cluster. If that says This Organization's Servers can't be reached, go to Organization, click Reset founding attempt, tick the confirmation and click Reset enrollment. The confirmation asks you to stop or erase the old Machine, but this server can keep running in its Cluster: the reset revokes Ployz Cloud's own access to it.

When the message is a new founding claim requires an uninitialized Server, you can add this server itself instead, once Ployz Cloud let go of the founding attempt or you reset it: run ployz server add --command --reset for a new token, and run the line it prints on the server. The reset takes the server out of its Cluster and removes every container Ployz runs on it, so its apps go down, and it can delete its mirrors, so don't use it while a mirror there may be the last copy of a volume.

If add says Server MY-SERVER belongs to another Cluster, the server is already in another Cluster, so this add stops before changing it. Remove it from that Cluster first, then add it here with a new token: signed in, run ployz server add --command and run the line it prints on the server. The token you used is tied to the server as it was, so rerunning the same line is refused.

If you can't manage that Cluster, signed in here run ployz server add --command --reset and run the line it prints on the server. The reset takes the server out of the other Cluster and removes every container Ployz runs on it, and it can delete its mirrors, so don't use it while a mirror there may be the last copy of a volume.

The reset can leave the server listed, as down, in the other Cluster:

  • If the server couldn't remove itself from its own copy of the Cluster's records, the add warns Server NAME (ID) reset, but couldn't confirm that its old Cluster removed it, then gives the reason and the ployz server forget ID --confirm dead line for the other Cluster.
  • If the server was cut off from the other Cluster while it reset, the add shows no warning, and the other Cluster keeps listing it.

Either way, whoever manages the other Cluster removes it there with ployz server forget <id> --confirm dead. In that Cluster, ployz server ls --json shows each server's id as machine.id.

The server may stay listed as down in the Organization you tried to add it to. ployz server ls --json shows its id as machine.id; forget it by that id, ployz server forget <id> --confirm dead, since a server you add again can share its name.

A forgotten server is refused when you add it again

Section titled “A forgotten server is refused when you add it again”

You'll see Server web-2 already participates, but Cloud doesn't list it as joined.

If an add of this server just stopped partway, run the same command again in a minute. If you forgot the server and it kept running, the token that add used is now tied to the server as it was before you forgot it. --reset gives the server a new identity, so that token won't work after a reset. Add it again with a new token. Signed in, either run ployz server add --command --reset and run the line it prints on the server, or run ployz server add root@HOST --reset to add it over SSH. The reset removes every container Ployz runs on that server and can delete its mirrors, so don't use it while a mirror there may be the last copy of a volume.

The server's Cluster is connected to another Organization

Section titled “The server's Cluster is connected to another Organization”

Adding your organization's first server, ployz server add can stop with:

error: This Server's Cluster is already connected to another Organization in Ployz Cloud. Contact Ployz support to move it.

If ployz server add was going to show you a recovery code, the first line is Recovery code not saved. Run ployz recovery rotate now to make one. instead, with the same cause. ployz server enroll --status shows Failed · and the same text without retry advice. With an older Ployz Cloud, the CLI still wraps this sentence in Cloud enrollment completion failed; rerun the same ployz server add command without --reset (keep all other options). and --status still advises running ployz server enroll again. Servers > Waiting to join shows Failed at Going live over that text.

An older ployz doesn't show Ployz Cloud's sentence. Its cause reads enroll HTTP 409: enrollment completion rejected, which it prints for other refusals too, so rerun the command once. If it fails the same way again, the server's Cluster may be connected to another Organization.

Ployz Cloud connects a Cluster to one Organization: the first one that added it. The server you're adding belongs to a Cluster another Organization already added, so this Organization can't manage it. Add a server that isn't in that Cluster instead, or manage it from the Organization that added it. On ployz.dev, to move the Cluster to this Organization, or if you don't recognize the other one, contact Ployz support. On a self-hosted Ployz Cloud, ask whoever runs it.

You'll see the joined Server did not become ready, or the first Server did not become ready on your first server.

If the cause says ployzd refused to start:, Ployz stopped waiting as soon as it read the refusal, over SSH or on the server itself. Go straight to step 3, then run the same command again.

If the wait printed The Server's daemon is running but still joining, or the error says the daemon stayed joining, the server started but couldn't catch up with your other servers. Start at step 2.

If it says Cloud candidate publication failed, the server started, but Ployz Cloud didn't see your other servers list it as joined before the wait ran out. Start at step 1.

  1. Run the same command again. A first start can take several minutes.
  2. Make sure your servers reach each other on UDP 51820, in both directions, in the server's own firewall and in any firewall your provider runs in front of them. On a server that runs firewalld, follow A server that runs firewalld doesn't answer your other servers.
  3. Read the server's logs with sudo journalctl -u ployz -n 200, or the error's cause: line when the wait reached the server over SSH. Only if they say refusing to change the retained WireGuard interface or refusing to replace the existing Docker network, follow Ployz stops over a leftover network. If they say UDP port 51820 is in use, follow Another program holds UDP port 51820.

A new server joined, but Global catch-up can't read another server

Section titled “A new server joined, but Global catch-up can't read another server”
Joined Server MY-SERVER.
error: Server joined, but Global catch-up is incomplete; it remains a Cluster member.
  cause: Global catch-up cannot plan from partial Service observations: web-1: Machine RPC payload failed: Could not decode the JSON payload: ...

The ployz that ran the join is older than web-1, and it can't read what web-1 runs. That's the new server's own ployz when it joined with a Fleet Token, or the computer where you ran ployz server add. Usually web-1 runs a service with a sealed variable, and that ployz is 0.3.3 or earlier. The message ends with restore peer connectivity and redeploy, but your network is fine. Releases after 0.3.5 say this ployz can't read web-1's answer, so update ployz on this host instead.

The server has joined and stays joined. To finish:

  1. Update ployz where the join ran to the release your other servers run, 0.3.4 or later. For the new server, upgrade Ployz on it.
  2. Redeploy, so its global services start on it.

A server waits for approval and never shows up

Section titled “A server waits for approval and never shows up”

A server you started with a Fleet Token runs ployz server checkin --install. That installs a service, ployz-checkin, which checks in with Ployz Cloud until you approve the server, and prepares Docker and ZFS meanwhile.

  1. Run ployz status on the server. The this server line says where it stands, and blocked lines say what stops it.
  2. Read the loop's log with sudo journalctl -u ployz-checkin -n 100, and preparation's with sudo journalctl -u ployz-prepare -n 200. A failed preparation is retried 10 minutes later, up to three times. Once you've fixed the cause, sudo ployz server checkin --install starts it again right away. Preparation that stopped mid-step on every try reads as failed. Preparation still running outside ployz-prepare is waited on, not retried or given up. That covers a run you started by hand with ployzd install --prepare, including a build that outlived the ployzd you ran. It needs a ployzd as new as the check-in loop, and /var/lib/ployz/prepare on ext2, ext3, ext4, XFS, Btrfs, tmpfs, overlayfs or ZFS. An older ployzd, another filesystem such as NFS, or a lock file check-in can't read leaves it judged by ployz-prepare alone.
  3. Check-in stopped means the loop isn't running, so nothing checks in and, a few minutes later, Ployz Cloud shows the server Offline. last answer is what it saw before it stopped, and checked in says when. sudo systemctl status ployz-checkin says why it stopped, and sudo ployz server checkin --install starts it again. Run ployz status with sudo: only root can tell whether the loop runs.
  4. Can't reach Cloud means the server can't reach Ployz Cloud over HTTPS, or something in between, such as a proxy, answered instead. It keeps trying every minute.
  5. Clock is off means Ployz Cloud refused the check-in and the server's clock is more than 4 minutes away from Cloud's, close to the 5 minutes Cloud allows, so the token isn't to blame. The reason line says by how much, and says too when Cloud refused the token itself: a token refusal from a server whose clock is that far off doesn't count against the token until the clock is right. Set the clock right, for example with timedatectl set-ntp true, and the running loop checks in again within a minute. If the install itself said so, it started nothing: fix the clock and run the install again.
  6. Not Ployz Cloud means the check-in URL reaches something that answers but isn't Ployz Cloud, such as a 404 or a web page. That can be a wrong PLOYZ_CLOUD_URL, or a proxy or firewall in between. The reason line names the URL. The loop tries again every minute. Point the URL at your Ployz Cloud, or let the server through, and run the install again.
  7. Held by Cloud means Ployz Cloud answered but held the check-in back, and the reason line gives Cloud's message. This Fleet Token already has N waiting servers. means the token has as many waiting servers as it allows. Rejected servers count too, so rejecting them frees nothing, and a server you add frees its place once Cloud's cleanup runs, at least a week after it joins. Run the install again with another token, or revoke the full one: that removes its waiting servers that haven't started joining and stops them checking in. Servers already joining finish, and joined servers stay. Otherwise Cloud is busy, and the loop asks again when Cloud says to.
  8. Needs a newer ployz means Ployz Cloud couldn't read this server's check-in. The loop tries again every hour. Install a newer ployz and run sudo ployz server checkin --install.
  9. Token refused means Ployz Cloud refused the Fleet Token once. The loop asks again once a minute has passed on Cloud's clock since that refusal, even across a restart, and goes on as before if Cloud accepts it. A new token in the file starts over with its own two chances. A refusal during the install itself ends it with an error instead, since the token was only just given, and starts nothing.
  10. Rejected, Token expired or Token revoked are final. The running check-in loop stops, deletes /etc/ployz/fleet-token, or the file you named with --token-file, if it still holds the token Cloud answered, and won't start again at boot. If the answer comes during ployz server checkin --install, the install ends with an error and keeps the file. A token Cloud refuses twice in a row, at least a minute apart by Cloud's clock, because you revoked or deleted it, reads Token revoked. If you put a new token in the file as the loop stops, it keeps it and restarts with it, unless Cloud rejected the server. Put a new Fleet Token in that file and run sudo ployz server checkin --install again to start a new check-in attempt. A non-rejection response clears the local rejection status, but Cloud can still reject the server. If the token file goes missing, the loop stops the next time it starts. Preparation already under way finishes.
  11. A blocked line saying ployzd ... is older than this ployz means you upgraded ployz but not ployzd. The loop keeps checking in with the older one, and Cloud shows the server as blocked until you run sudo ployz server checkin --install. It uses the first older one that answers, trying the one install chose, then PATH, then its own copy, and doesn't prepare Docker or ZFS with it. No ployzd means there is none at all: the loop exits until the same command installs one.
  12. Once you approve the server, the loop enrolls it and the this server line walks through Installing, Enrolling, Queued to join, Setting up ZFS, Joining and Publishing, the same stages Ployz Cloud shows, where Queued to join reads In line, Setting up ZFS reads Setting up storage and Publishing reads Going live. Queued to join means other servers are joining the fleet first. When it reads Joined, the running check-in loop turns itself off and deletes /etc/ployz/fleet-token, or the file you named with --token-file.
  13. Enroll failed means a stage failed, and the line after it names the stage and why, for example Enroll failed at enrolling: Docker is not running. The server goes back to waiting and Ployz Cloud shows the same failure. Fix the cause, then approve the server again in Ployz Cloud. If Cloud enrollment publication or completion failed, run ployz server enroll again, or use Enroll in Servers > Waiting to join. If the loop restarts mid-enroll, it picks up the same attempt rather than starting a second one and keeps the last reported stage. A resumed join that had committed and then fails reports the stage it had reached, such as joining, not the step it was retrying. If the join committed but a follow-up failed, the server stays Joined, the loop cleans up, and its log and status name that follow-up.
  14. Rejecting a server stops enrollment at the next safe point. A package install finishes before stopping at the enroll or queue step. Storage setup finishes once begun in the current run. After a restart from storage setup, rejection stops enrollment at the enroll or queue step. Once the daemon accepts the join and Cloud records Joining or Publishing, rejection lets enrollment finish, including after a restart. Cloud keeps its enrollment token valid and continues answering the attempt until it joins or fails. Expiring or revoking a Fleet Token lets an open enrollment finish.
  15. If Cloud selects a spare disk, the server builds its ZFS pool on that disk.
  16. When you check ports on waiting servers, each one listens on TCP 80 and 443 and UDP 51820 for at most 10 seconds, then closes them. Nothing stays listening after that. A check that arrives while ployz server checkin --install runs makes it pause for those seconds, and it says so. The server tells Cloud what it heard only after it closes the ports. One that restarts while it listens tells Cloud nothing for that check, so its ports read Unknown · server didn't report: check again. Open means Cloud reached the port and the server answered. Closed means the server listened but Cloud's connection was refused, so a firewall in front of it turns the port away. Couldn't listen · port in use means Ployz couldn't listen on that port in time. Usually another program, such as a web server, already holds it. Find it with sudo ss -lntup 'sport = :80', stop it, and check again. If nothing holds it, a delayed check-in reply or a busy server may have left too little time to listen, so check again. A server running a ployz older than this feature leaves every port Unknown.

A waiting server reads Blocked with UDP port 51820 is in use by avahi-daemon (pid 812)., or ployz status lists it under blocked. Ployz needs that port for WireGuard between servers, and while another program holds it, Ployz can't start on the server: an enroll would fail waiting for it. See what holds it with sudo ss -ulpn 'sport = :51820', then stop or restart that program. avahi-daemon, for one, can take the port at random and lets it go when restarted. The block clears on the server's next check-in.

A server that's already in a Cluster, or one you added with ployz server add, has no check-in. There, Ployz keeps restarting, and sudo journalctl -u ployz -n 50 says UDP port 51820 is in use by avahi-daemon (pid 812), so WireGuard can't listen, or by another program when it can't tell which. Free the port the same way, and Ployz starts on its next restart.

When ss lists the port with no program, another WireGuard interface, such as a tunnel set up with wg-quick, probably holds it. sudo wg show all listen-port names the interface. Move that tunnel to another port, or take it down.

Enroll refuses over an earlier enrollment's network

Section titled “Enroll refuses over an earlier enrollment's network”

You'll see refusing to enroll over an earlier enrollment's network. The server's Ployz state was wiped, but the ployz-wg interface or the ployz Docker network from its earlier enrollment is still there. Enroll stops before it changes anything. The message names each leftover, including the containers still attached to the network, and ends with the commands that clear them all. Removing those containers stops them, and their volume data stays on the disk. Run the commands, then run the enroll command again. After deleting /var/lib/ployz covers the old identity left in your server list.

sudo systemctl status ployz shows the service failed with status=78, and its logs say refusing to change the retained WireGuard interface or refusing to replace the existing Docker network. A ployz command that connects to that server over SSH says the same, after ployzd refused to start:. A command that goes through another server only says the server did not answer. Something else, or an earlier install, left a network Ployz can't prove is its own, so Ployz stops rather than take it over. It won't start again until you clear it. The log line ends with the safe recovery for your case.

For a leftover ployz-wg interface:

sudo systemctl stop ployz.socket ployz
sudo ip link delete ployz-wg
sudo systemctl reset-failed ployz.socket ployz
sudo systemctl start ployz

For a Docker network named ployz, stop Ployz the same way. Run docker network inspect ployz and find its owner from its labels and attached containers. Move or remove those containers through whatever deployed them. Once the network is empty and nothing needs it, run docker network rm ployz, then the last two commands above.

A server can have both. Clear them together, because Ployz names only the first one it meets on each start.

/var/lib/ployz holds the server's identity, and the ployz-wg interface and ployz Docker network belong to that identity. Deleting the folder doesn't remove them, so the server's next enrollment refuses over them. If you delete /var/lib/ployz, clear them too, before you add the server again:

sudo systemctl stop ployz.socket ployz
sudo ip link delete ployz-wg
docker network inspect ployz --format '{{range .Containers}}{{.Name}} {{end}}'
docker rm -f MY-CONTAINER ...
docker network rm ployz
sudo systemctl reset-failed ployz.socket ployz
sudo systemctl start ployz.socket

The containers that docker network inspect lists are the apps and the ingress proxy the server ran before. Removing them stops those apps; their volume data stays on the disk.

Your server list still shows the old identity as a second row with the same name, so the join can end with Global catch-up is incomplete and MY-SERVER: no terminal response. The server has joined all the same. Forget the old row by its ID, which ployz --json server ls shows, as in Forget a server that is gone. Then redeploy, so your apps and the ingress proxy start on the server again.

You'll see This Organization's Servers can't be reached when you add another server, and an alert on the Organization page. The alert also shows while your first server installs, so give it a few minutes first.

No other server can join until the first one finishes.

  1. If the first server still exists, run its command on it again. If its token expired, run ployz server add --command for a new one; it picks up where the server left off.
  2. If that server is gone, or you want to start over, make sure it's stopped or erased at your provider. Then go to Organization, click Reset founding attempt, tick I confirm the old Machine has been stopped or erased. and click Reset enrollment. Tokens you made for other servers keep working. If the reset says to use Forget Servers, go to Organization and, under Danger, click Forget servers, or run ployz server forget --confirm <your organization's slug> where you're signed in. Forget deletes every enrollment token of your organization, so make a new one for each server, with ployz server add --command or Add servers.
  3. Add a server again.

If your first server did finish joining, there's no founding to reset, and the CLI says so. To start over, stop or erase your servers, then forget them as in step 2.

The Servers page shows Can't reach your servers right now, with what your servers last reported or nothing at all. Adding a server can fail with This Organization's Servers can't be reached. A CLI command, or a deployment that couldn't run, says Server web-1 isn't answering, naming each server it couldn't reach by the name it last reported, or by its ID if Ployz never learned one. When the CLI asked through Ployz Cloud, it says Server web-1 isn't answering through Ployz Cloud and asks you to check that the server is online, its Ployz daemon is running, and its outgoing https isn't filtered: any of the three looks the same from the CLI. When a server answered but refused every connection, the error names the server tried last, as in Server web-2, tried last, refused the connection. When its connection finds no server answering, ployz status lists Server web-1 isn't answering under needs attention, and ployz deploy --plan warns first that the deploy can't run until it answers. A deploy whose web traffic check couldn't reach a server warns Couldn't check ingress: Server web-1 isn't answering, with what the connection said in parentheses, and exits 3. A server that stopped answering because ployz server rm was removing it doesn't count: the removal says so in the server's own record before it stops answering (Ployz 0.3.4 or later on that server and on the server the CLI connects through). Any deploy step a server leaves unanswered reads Server web-1 did not answer during create Container, with what the connection said beneath it. A server that just joined or restarted answers within a minute; try again then. While your organization's first server is still enrolling, the page shows its enrollment progress instead: there's no server to reach until it joins.

  1. Check at your provider that the servers are running.
  2. On a server, run sudo systemctl status ployz. If Ployz is stopped, start it with sudo systemctl start ployz, and read its logs with sudo journalctl -u ployz -n 200.
  3. If your servers' outgoing traffic is filtered, allow https to relay.ployz.dev. When Cloud can't reach your first server, adding it fails with Cloud can't reach this Server through the relay or directly. Allow https to relay.ployz.dev from the server, and from Cloud if you host it yourself, then try again.
  4. If you deleted the servers, see Forget servers you deleted.
  5. If the servers run Ployz 0.2 or 0.3.0-beta.6 and earlier, Cloud can't reach them through the relay. Upgrade one over SSH, then the rest from Cloud, as in Servers on 0.2 or 0.3.0-beta.6 and earlier. A server on 0.3.2 or earlier is used only through a newer one. Upgrade it as in Servers on 0.3.2 and earlier; it is used once a server that holds your cluster's key is back online.
  6. If the page reads Waiting for a server that holds your cluster's key, a server answers but doesn't hold your cluster's key yet. It's used once a key holder is back online. Keep it running, and don't forget it. If the page stays this way while a server that holds the key is online, check the daemon logs on the server that answers: sudo journalctl -u ployz -n 200.

The CLI says a server runs 0.2 or 0.3.0-beta.6 and earlier

Section titled “The CLI says a server runs 0.2 or 0.3.0-beta.6 and earlier”
This Server runs Ployz 0.2 or a 0.3.0 beta up to beta.6, which this release can't reach through the relay.

Upgrade one server over SSH, then the rest from Cloud, as in Servers on 0.2 or 0.3.0-beta.6 and earlier.

A server that answers but refuses says so instead, with its own reason: Server web-1 couldn't give this command access: Cluster store is not available. It is running, so don't forget it. Read why in its logs with sudo journalctl -u ployz -n 200.

error: Corrosion HTTP request failed, with the cause Connection refused, means the server answered but its cluster store wasn't running. If it was stopped, as by docker stop ployz-corrosion, Ployz starts it again within a few seconds, including while the daemon is still starting; try again then.

error: this Server's Cluster store holds no other Server: it was recreated and can't reach them yet

A server keeps the cluster's shared state in /var/lib/ployz/corrosion/store.db. If that file is lost, for example after a power loss or a manual wipe, Ployz starts an empty one. The server keeps its tunnels to the other servers, and the empty store refills from them, usually within a minute. Until then, commands through that server fail with this error. Commands through another server still work.

  1. Wait a minute and try again.
  2. If it persists, run ployz server ls through another server. If this one shows down there, make sure UDP 51820 is open between your servers, then restart Ployz on it with sudo systemctl restart ployz.

If you still have the old store.db, you don't need it back: the refilled store holds the same state.

The server's page says Can't reach web-2. Services that run only there are down.

Ployz Cloud also shows a server Offline within about ten seconds of Ployz stopping on it, for example after sudo systemctl stop ployz.socket ployz, because nothing there answers for what it runs. Its containers read Server offline rather than Crashed, since what they last reported may no longer be true. Start Ployz again and the server reads online. Such a stop doesn't remove the server: it keeps its place among your servers until it reboots or you remove it.

  1. Check at your provider that the server is running, then run sudo systemctl status ployz on it.

  2. Make sure UDP 51820 is open between your servers.

  3. If the server is gone for good, forget it, then deploy again. Its page offers Forget server while it's offline.

    See When a server goes down for which services come back, and When its server goes down to bring a volume back from its mirror with Use mirror before you deploy.

Ployz reads a server's public IP and network addresses again each time Ployz starts on it. When a server comes back from a stop with a new address, the other servers learn it, and it shows in ployz server ls. If the public IP lookup fails, the server keeps its old address and logs a warning; restart Ployz on it with sudo systemctl restart ployz to try again.

An address you chose yourself never changes on its own. That means --public-ip or --wg-endpoint given to ployz server add or ployz server set. To change it, run ployz server set again.

A server on 0.3.0-beta.6 or earlier keeps the public IP and network addresses it has when you upgrade it to a release after 0.3.0-beta.7. Those older releases didn't record which addresses you chose, so Ployz treats both as chosen. From then on the server doesn't pick up a new address on its own: after its IP changes, run ployz server set. Only resetting the server goes back to automatic addresses. A server already upgraded to 0.3.0-beta.7 may have swapped an address you chose for one it found; set yours again with ployz server set.

If every server's address changed at once, the servers can't reach each other to share the new addresses, and each one shows the others as unreachable. Tell them where one of them is.

Servers lost each other after their IPs changed

Section titled “Servers lost each other after their IPs changed”

Every server got a new public IP address at once, for example after a provider moved or restarted them. Each server shows itself running and the others as not answering, in ployz server ls and in Ployz Cloud, because each one still dials the others at their old addresses.

Tell the servers where one of them is now. Run this once, with that server's name and its new public IP address:

ployz server set web-2 --wg-endpoint 203.0.113.20

Every other server the command reaches starts dialing web-2 at that address right away; WireGuard uses port 51820 unless you name another, as in 203.0.113.20:51821. Once one of them reaches web-2, web-2 answers from its own new address and the two find each other, usually within seconds. The command says which servers took the address, then saves it on web-2:

web-1 now dials web-2 at 203.0.113.20:51820.
Updated Server web-2.

If web-2 still can't be reached, the second line reads ! web-2 didn't answer, so its record may or may not have changed. and the command still succeeds. Run it again once both servers show up in ployz server ls to save the address on web-2 too. With more than two servers, repeat the command for any server that still shows as not answering.

Run it from your computer with Ployz Cloud, or on one of the other servers itself. Run on web-2, it only saves the address on web-2.

You'll see ! Large packets to web-2 are dropped on its mesh path from the Server that answered, so calls with large replies to web-2 fail. in ployz server ls. Small requests still reach web-2, so it answers pings, but its rows may read Volume support unknown or did not answer. A deploy says ! Server web-2: deployment observations are incomplete (containers: target Machine RPC timed out; large packets to web-2 are dropped on its mesh path from the Server this command entered by) and leaves web-2 as it is. From 0.3.8 it deploys on the other servers and exits 3, as when a server is down; the replicas on web-2 keep running until a deploy after the path heals replaces them. Before 0.3.8, every deploy that started a replica failed on every server, with list Containers failed, Machine web-2 did not answer.

While the path stays broken, the confirmations from web-2's proxy that a deploy waits for may stop arriving. A server that reads down no longer counts toward that wait. Otherwise, after 5 minutes, the deploy fails with proxy on web-2 did not confirm; large packets to web-2 are dropped on its mesh path from the Server this command entered by.

ployz server show web-2 still shows it. When web-2 doesn't answer through the server the command entered by within 3 seconds, the CLI reads it over its own connection and prints ! web-2 did not answer through the Server this command entered by, so it was read over its own connection. This works signed in to Ployz Cloud, or with a context that lists web-2. Without a connection of its own, the command fails with large packets to web-2 are dropped on the mesh path from the Server that routed this call, or target Machine RPC timed out if it gives up first; the warning above still names the cause.

Something between your servers, often a provider's network or a VPN with a smaller packet size, drops packets above a size. From 0.3.7, each server checks every minute that a full-size packet reaches each other server. When it doesn't, the server finds the largest size that does and sends to that server in packets of that size, so nothing fails. ployz server show on that server then lists the peer with its smaller size, as in peer = web-2 0.4 ms, mtu 1310; a peer at full size shows no size. The server tries full size again every 10 minutes and goes back to it once the path carries it, logging the mesh path to web-2 carries 1420-byte packets again.

The warning above means even 1280-byte packets, the smallest IPv6 allows, are lost, so Ployz can't work around it. Ask your provider why UDP packets on port 51820 above that size are dropped between the two servers, or move one of them. Once full-size packets cross again, the warning goes away within a minute. A server on 0.3.6 or earlier doesn't check, so it can't name the cause; a call routed through it says target Machine RPC timed out instead.

Services on other servers stop answering after a firewall change

Section titled “Services on other servers stop answering after a firewall change”

Ployz keeps its own iptables rules first in INPUT and Docker's DOCKER-USER chain. A tool that later inserts rules above them or flushes them, such as ufw-docker, firewall-cmd --reload or a security agent, can cut services off from other servers. Ployz applies its rules again every minute, so traffic returns within a minute with no restart.

When a rule Ployz keeps first was missing or no longer first, journalctl -u ployz names it, as in restored firewall rules that were missing or no longer first: filter DOCKER-USER: -i ployz-wg -o ployz -j ACCEPT. The line also names the TCP clamp on ployz-wg when it was missing. It doesn't name rules inside Ployz's own PLOYZ- chains, which Ployz rewrites whole each minute, or its NAT rule, which Docker moves down every time it creates a network or restarts. When the line repeats every minute, something keeps rewriting the firewall. Find it and have it add its rules below Ployz's.

If iptables fails, for example because another program holds its lock for more than 10 seconds, the log says failed to reapply firewall rules with iptables' reason, and Ployz tries again a minute later. Ployz 0.3.16 and earlier put the rules back only when the server restarts.

A server that runs firewalld doesn't answer your other servers

Section titled “A server that runs firewalld doesn't answer your other servers”

A command or deploy says ! web-2 did not answer; its rows are missing. with cause: Permission denied (os error 13), and a deploy leaves an old replica running on another server. Check whether firewalld runs on that server:

systemctl is-active firewalld

If it says active, firewalld refuses the calls your other servers make to Ployz on that server, even though Ployz opens its ports in iptables. List its zones, and note the one shown above its public interface, such as eth0:

sudo firewall-cmd --get-active-zones

Then open UDP 51820 in that zone, shown here as public, and trust ployz-wg, the interface your servers reach each other on:

sudo firewall-cmd --zone=public --permanent --add-port=51820/udp
sudo firewall-cmd --zone=trusted --permanent --add-interface=ployz-wg
sudo firewall-cmd --reload

ployz-wg only carries traffic from your own servers and the apps on them. Trusting it lets them reach anything listening on this server's private address, as they could without firewalld, while Ployz keeps its own ports closed to everything else. On some older systems the reload also clears Ployz's own iptables rules; Ployz puts them back within a minute, as Services on other servers stop answering after a firewall change describes. On Ployz 0.3.16 and earlier, run sudo systemctl restart ployz after the reload instead.

Apps see IPv6 visitors as 10.210.x.1, or services can't reach IPv6 addresses

Section titled “Apps see IPv6 visitors as 10.210.x.1, or services can't reach IPv6 addresses”

Through a domain, your app's X-Forwarded-For should show each visitor's own address, and a service that publishes no port on the server should reach IPv6 addresses. If X-Forwarded-For shows your server's private gateway, such as 10.210.0.1, for visitors over IPv6, or a container Ployz 0.3.6 or later created for a service that publishes no port has no IPv6 address, the server's container network has no IPv6. journalctl -u ployz says why:

  • learn IPv6 from router advertisements: the server takes its IPv6 route or address from router advertisements, which IPv6 forwarding would stop. Run the sysctl -w command the log names, add the same setting to a file in /etc/sysctl.d/ so it survives a reboot, then run sudo systemctl restart ployz.
  • /etc/docker/daemon.json sets "ip6tables": false or dockerd runs with --ip6tables=false: Ployz needs Docker's default iptables firewall with ip6tables on. The line names the setting and where it found it, daemon.json or Docker's command line. Remove it there, restart Docker, then run sudo systemctl restart ployz. "iptables": false and "firewall-backend": "nftables" read the same way. Restarting Docker is enough, with no reboot.
  • /etc/docker/daemon.json changed at …, after dockerd started at …: Ployz can't tell what the running Docker loaded, so the network stays IPv4-only until Docker restarts. With no daemon.json, it reads /etc/docker changed at … and holds no daemon.json, as after deleting the file. Run sudo systemctl restart docker, then sudo systemctl restart ployz. systemctl reload docker isn't enough. Other lines name what Ployz couldn't read, such as daemon.json itself, Docker's pidfile, or more than one dockerd, and the step that clears it.
  • the IPv6 mesh fence failed: the log line gives ip6tables' reason. Ployz tries again at its next start.
  • could not take IPv6: the log line gives Docker's reason. Ployz tries again at its next start.

A port a service publishes on the server always shows IPv6 visitors as the gateway, and a service that publishes one has no IPv6 address. Give the service a domain instead to see visitors' addresses and reach IPv6 addresses.

When the services have an IPv6 address but can't reach IPv6 addresses, check the server itself: curl -6 https://example.com on it fails when the server has no IPv6 route out. Its container network can still take IPv6, but nothing carries it further. Ask your provider for IPv6. Until then your services keep reaching IPv4 addresses.

When only some services can't reach IPv6 addresses, check whether each has an IPv6 address: ployz exec SERVICE -- cat /proc/net/if_inet6 lists one starting fdcd when it does, if the image has cat. A container created before 0.3.6 has none until Ployz replaces it. A deploy that changes the image or a variable replaces it. A deploy that changes nothing, or only CPU or memory limits, keeps it.

A service on the server gets NXDOMAIN or a timeout for a name like postgres.app.internal, while ployz service ls shows the target running.

  1. Run sudo systemctl status ployz-dns. Its Status line says serving, loading, or waiting for the Corrosion token while it works. idle means it has no names to answer. Either the server isn't in an organization yet, or it couldn't read its list of names. In the second case its logs say why, and it starts answering once the list is readable.
  2. If the Status line says waiting for port 53 on 10.210.0.1 (your server's private address), another program holds that port, often dnsmasq from libvirt or a local DNS cache. Find it with sudo ss -lntup 'sport = :53', then stop it or make it listen only on other addresses. Private DNS starts answering on its own once the port is free.
  3. Read its logs with sudo journalctl -u ployz-dns -n 200.
  4. If the unit is missing, run sudo systemctl restart ployz. Ployz recreates it.

The bottom bar shows Add a server instead of Deploy

Section titled “The bottom bar shows Add a server instead of Deploy”

Your organization has no server, so nothing can deploy.

Add a server with ployz server add. Your projects and settings are kept. Once it joins, Deploy is back and Ployz deploys your environments to it. Staged changes stay in the bottom bar.

See what uses the space:

df -h /
sudo docker system df
  • Build cache. Run sudo ployz server build-cache-clear on the server. It keeps your images and refuses while a build runs.
  • Images from a registry, like an old postgres version, stay until you remove them with sudo docker image rm. Ployz cleans up the images it builds on its own.
  • Container logs. Ployz sets each container's log settings itself: Docker keeps up to three 10 MB files per container and rotates them. log-opts in /etc/docker/daemon.json doesn't change them. A container started before your server ran this release keeps its old settings until its next deploy.
  • Plain Docker volumes have no size limit, and docker system df above counts them.
  • Kept container logs and metrics. Ployz keeps the logs of containers it ran, even after they're replaced, and the server's metrics in ployz-observe under Docker's directory. docker system df doesn't count them; sudo du -sh /var/lib/docker/ployz-observe does. Together they're capped at 5% of the disk, kept between 512 MB and 5 GB. Logs go after 30 days. When free space drops below 10% of the disk (at least 2 GB), the oldest logs and metrics go first until it's back.

A deploy says there isn't enough disk space

Section titled “A deploy says there isn't enough disk space”

You'll see Not enough disk space: about ... more free space is needed.

A managed volume needs its whole storage limit free on the server, on top of the quarter of the disk Ployz keeps for the system. Free disk space, give the server a bigger disk, or lower the volume's storage limit before you deploy it. See Volumes.