ProxPatch: Automated rolling updates for Proxmox clusters

Patching a single Proxmox node is just apt full-upgrade and a reboot. Patching ten clusters of five to eight nodes each means repeating the same operation fifty times: drain the node, migrate the VMs, update, reboot, wait for quorum to return, move on to the next. It’s long, mechanical, and exactly the kind of task where you end up forgetting a step.

ProxPatch automates this grind. It’s a tool written in Rust by gyptazy (the author of ProxLB ), and it does one thing: rolling updates for a Proxmox VE cluster, without interrupting the workloads that are running.

The problem it solves#

When you maintain multiple clusters with many nodes, manual updates aren’t a technical challenge: they are a problem of time and consistency. Each node requires the same sequence, and the more nodes there are, the higher the probability of triggering two reboots before quorum has returned. That’s how you break a cluster.

ProxPatch removes that burden. You install it, enable it, and it runs the correct procedure on each node, in order, validating the cluster state between each step. The benefit isn’t raw speed (it takes its time, by design), it’s that you no longer have to babysit the operation.

How it works#

The flow is deliberately simple and auditable. No database, no API token, no orchestration framework: it only relies on Proxmox’s native tools (pvesh, qm, SSH) and the existing SSH trust relationship between cluster nodes.

On each cycle, ProxPatch runs two passes:

Pass 1 — Preparation (all nodes, no reboot)#

• Inventory via pvesh: node resources, running VMs and their memory consumption.
• It’s not just a simple apt update: it’s an apt dist-upgrade over SSH on each node: apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade.
• Detection of the /var/run/reboot-required flag to know if a kernel (or something else) requires a reboot.

Pass 2 — Rolling reboot (only the nodes that need it)#

• For each node that requires a reboot: compute a migration plan, then live-migrate the VMs (pvesh create /nodes/NODE/qemu/VMID/migrate -target TARGET -online 1 -with-local-disks 1).
• Health check: all nodes online and quorum healthy. If the cluster is degraded, ProxPatch stops without rebooting.
• Reboot the node, wait for it to come back, verify rejoining the cluster, another health check before moving to the next.

If ProxLB is installed, it is stopped during patching and restarted at the end, to prevent it from triggering a rebalance in the middle of the operation.

Prerequisites: a PVE 8.x or 9.x cluster, at least three nodes, quorum maintained throughout the operation, shared storage (Ceph, NFS) for live migration, passwordless SSH key access between nodes, and jq installed on the node running ProxPatch.

One thing to know: ProxPatch is a daemon. Once the systemd service is started, it automatically loops every six hours. The interval is hardcoded in the binary and is not configurable for now (neither in the config file nor via a flag). It’s not a cron you trigger; it’s a process that patches continuously. Keep that in mind for what follows.

Installation#

ProxPatch is installed on a single node of the cluster. From that node, it discovers the others via SSH and orchestrates everything remotely. Do not enable the service on multiple nodes at the same time: two simultaneous instances on the same cluster means guaranteed outage.

Online (recommended): APT repository#

# Add the official gyptazy repository
curl https://git.gyptazy.com/api/packages/gyptazy/debian/repository.key \
  -o /etc/apt/keyrings/gyptazy.asc

echo "deb [signed-by=/etc/apt/keyrings/gyptazy.asc] \
  https://packages.gyptazy.com/api/packages/gyptazy/debian trixie main" \
  | tee -a /etc/apt/sources.list.d/gyptazy.list

apt-get update
apt-get install -y proxpatch

Replace trixie with bookworm depending on your Proxmox version.

Offline (airgap): .deb package#

On an Internet-isolated cluster, retrieve the .deb from https://cdn.gyptazy.com/debian/proxpatch/ or from the ProxPatch GitHub repository on a connected machine, transfer it to the target node, then run the installation:

# from a machine with Internet access
wget https://cdn.gyptazy.com/debian/proxpatch/proxpatch_0.1.2_amd64.deb

# transfer to the target node
scp proxpatch_0.1.2_amd64.deb root@pve-node01:/root/

# install on the node
dpkg -i proxpatch_0.1.2_amd64.deb

The package depends on jq. On Proxmox it is not always present, and in a disconnected environment dpkg won’t pull it automatically. Bundle it as well (apt-get download jq on the connected machine, then dpkg -i), or check beforehand (which jq).

A configuration?#

In the standard case, no configuration is needed: ProxPatch connects as root via the cluster’s SSH trust and discovers the nodes on its own.

If you don’t want to use root, you can create /etc/proxpatch/config.yaml:

ssh_user: proxpatch
deactivate_proxlb: true

With an ssh_user defined, commands go through sudo. The user must therefore have passwordless sudo and SSH access to all nodes. A minimal sudoers entry:

User_Alias  PROXPATCH = proxpatch
Cmnd_Alias  PROXPATCH_CMDS = \
    /usr/bin/pvesh create *, \
    /usr/bin/apt-get update, \
    /usr/bin/apt-get -y dist-upgrade, \
    /usr/sbin/reboot, \
    /sbin/reboot
PROXPATCH ALL=(root) NOPASSWD: PROXPATCH_CMDS

Note a detail: since 0.1.2, the systemd unit does not load a config file by default (ExecStart=/usr/bin/proxpatch, without -c). Creating the YAML is therefore not enough; you must point to it. The cleanest way is a drop-in:

systemctl edit proxpatch

[Service]
ExecStart=/usr/bin/proxpatch -c /etc/proxpatch/config.yaml

Sensitive clusters: disable automation#

This is the important point. ProxPatch is still young (0.1.x versions) and the author himself describes it as experimental, to be validated in a lab before any production use. Combined with the fact that it’s a daemon that continuously patches every six hours, that means a node can update and reboot without human intervention.

For a homelab or a test cluster, that’s exactly what you want. ProxPatch is great for dev and QA clusters: it automates rolling updates and lets you see if everything works well before going to production. For a critical cluster, no. On those clusters, install the package to have the binary available, but disable automation:

systemctl disable proxpatch && systemctl stop proxpatch

This way you stay in control: you launch ProxPatch manually when you have scheduled the maintenance, you watch it, and it never wakes up on its own to reboot a node at 3 a.m. on a production day.

# one-off manual launch, in debug mode to follow the flow
/usr/bin/proxpatch -d

Note: even when launched directly, the binary enters its loop (6 h by default). For a strictly one-off run, monitor and interrupt the process once the cycle is complete.

In summary#

ProxPatch does one thing well: it performs a correct rolling update on a Proxmox cluster, migrating VMs and checking quorum between each reboot. On a fleet with many nodes, that’s less time and less mental load.

Three reflexes to keep in mind:

• install it on one single node per cluster
• on critical clusters, run systemctl disable proxpatch && systemctl stop proxpatch to keep triggering under human control
• regularly reboot your nodes to benefit from kernel updates, depending on the criticality of your infrastructure and security updates

Today, I use ProxPatch on my personal cluster and on a few test and development clusters at work. I haven’t encountered any major issues with ProxPatch. I’ve gained time and peace of mind. Instead of spending a day managing updates manually, I can focus on other tasks while my machines update without my intervention.

Sources#

• https://github.com/gyptazy/proxpatch