Reboot your Proxmox servers without breaking Ceph

In a Proxmox cluster with distributed storage using Ceph , it’s important to follow certain preparation steps to avoid damage. The procedure below has been run many times on different Proxmox clusters (versions 5 to 9) between 2020 and 2026, with Ceph versions 14 to 20 (Tentacle).

First of all, don’t forget to back up your data regularly and to test the restore. The procedure below is not foolproof, so take your precautions. Many thanks to Matthieu , with whom I discovered and validated this procedure back in 2020-2021.

Prepare Ceph: enable “global flags”#

To enable the flags (which are options for the Ceph cluster), you can choose between the command line and the Proxmox web interface. Do all the steps with a single method, either through the web interface or via the command line, to avoid conflicts or redundant actions.

Proxmox web interface#

From the web interface of one of your cluster’s Proxmox nodes, enable the following two global flags, in the menu “Ceph > OSD > “Manage Global Flags” button:

• noout: prevents Ceph from marking unavailable OSDs as “out”, thus avoiding triggering a massive redistribution. This is necessary because when servers are powered off, their OSDs are offline.
• norebalance: disables the redistribution of PGs (Placement Groups) across OSDs. With fewer OSDs available, Ceph would automatically rebalance the PGs, which would cause unnecessary load and an increased risk in case of storage over-provisioning.

Note: enabling these options will put your Ceph cluster into a “HEALTH_WARN” state to signal both the presence of these options and the unavailability of the OSDs/hosts. This information is not critical on our end; feel free to check the Ceph logs during the maintenance operation to make sure no other alert slips in at the same time.

Command line#

Via SSH, log in as root on a Proxmox server in your cluster. Two commands are to be run on a single node of the Proxmox cluster to enable these flags across your entire Ceph cluster: ceph osd set noout and ceph osd set norebalance. The options produce no output, so run the ceph status command to see the state of the Ceph cluster. It should now be in the “HEALTH_WARN” state.

Reboot procedure#

Since Proxmox does not (yet) have a built-in maintenance mode, proceed as follows:

1. Reboot the Proxmox servers one by one if you have small clusters (3 to 8 nodes), either by clicking the “reboot” button from the web interface, or via SSH by typing reboot. Don’t reboot all the servers at the same time, you risk breaking the quorum (corosync tool) and bringing down your infrastructure. Wait until you see the rebooted server back “up” in the web interface (or via the pvecm status command line).
2. After each reboot, verify that all the OSDs of the rebooted server are operational (each OSD must have the status “up” and “in”). You can see this information either from the web interface (in Proxmox, select a server, then click “ceph” > “osd”), or via the command line ceph osd tree.
3. Repeat steps 1 and 2 for each server to reboot.

Finalization#

The operation can be tedious if you have Proxmox clusters with many servers. When you have several dozen machines, you can reboot multiple nodes at the same time, but keep a close eye on the state of corosync and ceph.

Once all the Proxmox servers have been rebooted, disable the “global flags”. As at the beginning of the article, perform this either through the web interface or via the command line by typing ceph osd unset noout then ceph osd unset norebalance.

Monitor the state of the Ceph cluster while it performs a quick rebalance, via the ceph status command. The information is also visible in the Ceph dashboard of the Proxmox web interface. At the end of the procedure, you should return to a “HEALTH_OK” state.

This procedure should be performed regularly to keep your clusters up to date. The VMs will keep running normally, with no service interruption. Once the rebalance is finished, the cluster returns to a stable state with no data loss or unnecessary load.