Building a Nested Proxmox VE + Ceph Lab

Ever wanted a full Proxmox VE cluster with Ceph storage, but only have one machine? This guide shows how to set up a 3-node Proxmox VE cluster with integrated Ceph storage, all nested inside a single physical host. In other words, we will install Proxmox VE on a powerful PC, create three Proxmox VM instances on it, cluster those VMs, and deploy Ceph across them. Each VM will run Ceph monitor/manager and OSD services, simulating a small hyper-converged cluster. The result is a complete Ceph-backed Proxmox environment perfect for home lab experimentation and learning, without needing three separate servers.

The high-level architecture is: one physical host (running Proxmox) with two bridge networks, and three nested Proxmox nodes (pve-1, pve-2, pve-3) as VMs. Each VM has an OS disk plus extra virtual disks for Ceph OSD storage. We will separate the network traffic so that Proxmox cluster communication (Corosync) is on one subnet, and Ceph replication/public traffic is on another. This isolation prevents Ceph’s heavy I/O traffic from interfering with the latency-sensitive cluster heartbeat.

Hardware and Host Preparation

Building this nested lab requires a fairly powerful host machine. Proxmox with nested VMs and Ceph will demand significant CPU, memory, and disk I/O resources. As a baseline, consider the following minimum specs:

• CPU: 8-core / 16-thread Intel or AMD processor with virtualization support (VT-x/EPT or AMD-V). More cores are better for handling multiple VMs.
• Memory: ≥ 32 GB RAM. This allows each nested node to have around 8–10 GB, with some left for the host. (If you have more RAM, allocate more to VMs for Ceph’s use.)
• Storage: Fast SSD or NVMe drives. For example, use a 500 GB NVMe for the Proxmox host OS and ISO storage, and add 2–3 SSDs (around 500 GB–1 TB each) dedicated as Ceph OSD disks. Ceph performs best with SSDs in small clusters, and all-flash storage will significantly improve I/O and recovery times in a 3-node lab. Avoid slow HDDs, as they will bottleneck the cluster.
• Network: At least one 10 Gbps network interface (or two 1 Gbps NICs if 10G is unavailable). This setup will use VLANs or separate NICs to carry the two networks (cluster vs Ceph). A single 10 GbE link can handle both networks via VLAN tagging. (In our lab, we use one 10 GbE port split into VLAN 100 for the Proxmox cluster and VLAN 101 for Ceph public traffic.) Note: Ceph can saturate 1 Gbps easily; using 10 GbE or higher is recommended even for small clusters.

Example Host Configuration: For instance, a suitable host could be a workstation-grade PC (e.g. Intel Xeon or AMD Ryzen) with 12 cores / 24 threads, 64 GB RAM, a 500 GB NVMe boot drive, and three additional SATA or NVMe SSDs for Ceph. One 10 GbE NIC (or dual 1 GbE) provides networking. In our example, we trunked two VLANs on the 10 GbE NIC: one for internal cluster communication and one for Ceph storage traffic.

Installing Proxmox on the Host

First, install the latest Proxmox VE 8.x on the physical machine as you normally would. Ensure that virtualization extensions are enabled in your BIOS/UEFI (Intel VT-x/VT-d or AMD SVM). Once Proxmox is installed on the host, a few extra configurations are needed to support nested virtualization and networking:

• Enable Nested Virtualization: By default, Proxmox does not pass through virtualization extensions to its VMs. Enable this so that our nested Proxmox VMs can run their own guests (Ceph daemons don’t need it, but Proxmox inside the VM might run LXC or test VMs). For Intel CPUs, edit /etc/modprobe.d/kvm-intel.conf to add:

options kvm-intel nested=Y

• For AMD CPUs, use /etc/modprobe.d/kvm-amd.conf with:

options kvm-amd nested=1

• Reboot the host after adding this. Then verify nested virtualization is enabled by running:

cat /sys/module/kvm_intel/parameters/nested

It should output Y. (Use kvm_amd for AMD.) This confirms the host will expose the VMX/SVM feature to guest VMs.

• Configure Network Bridges: Next, set up network bridges on the Proxmox host for our two networks. In the Proxmox web UI on the host, go to Datacenter → → Network. Create a Linux bridge for the Proxmox cluster traffic:
  • Bridge vmbr0 attached to the physical NIC (e.g. enp3s0). Assign it an IP address on the cluster LAN (for example, 192.168.100.1/24). This network will be used for Proxmox management and Corosync cluster communication.
  • For Ceph, create a second bridge vmbr1. If you have a second NIC, attach vmbr1 to that; otherwise you can create vmbr1 on a VLAN sub-interface of the same NIC (e.g. enp3s0.101 for VLAN ID 101). Give vmbr1 an IP on a different subnet (e.g. 192.168.101.1/24). This will serve as the Ceph public network. Separating Ceph traffic from Proxmox’s cluster network is highly recommended – it prevents Ceph’s high-bandwidth replication and client I/O from disrupting the low-latency cluster heartbeat and API traffic. Apply these network settings and ensure the host can communicate on both networks.

At this point, the physical host is ready with Proxmox installed, nested virtualization enabled, and two networks (bridges) configured.

Creating Nested Proxmox VM Nodes

With the host prepared, the next step is to create three Proxmox VE virtual machines which will act as our cluster nodes (pve-1, pve-2, pve-3) inside the host. Use the Proxmox web UI on the host to create these VMs with identical settings (besides name and IP). Key configuration points for each VM:

• Name & OS: Create a VM and name it pve-1 (and similarly pve-2, pve-3 for the others). Mount the Proxmox VE 8.x installer ISO on each VM (upload the ISO to the host’s local storage if you haven’t already).
• System & CPU: Use BIOS (SeaBIOS) or UEFI (either works). Assign 2–4 vCPUs to each VM. Set the CPU type to “Host” so the VM can leverage the host’s CPU features. This improves performance in nested virtualization by exposing all CPU extensions to the guest.
• Memory: Allocate around 8–10 GB RAM to each VM. This gives Ceph and the Proxmox services in the VM enough memory. (If your host has more RAM, you can increase this. Using the balloon driver is optional for memory flexibility.)
• Disks: Give each VM a boot disk of about 40–50 GB (virtio SCSI or IDE, stored on the host’s local-lvm). This will hold the Proxmox OS for that VM. Add two additional disks to each VM (virtio or virtio SCSI, raw format recommended) of, say, 100 GB each – these will serve as that node’s Ceph OSD drives later. Ensure these disks are unformatted (don’t create a filesystem; Proxmox’s Ceph setup will use the raw block devices).
• Network: Add two network adapters to each VM. Use the VirtIO model for best performance. Attach one NIC to vmbr0 (this puts the VM on the Proxmox cluster network) and the second NIC to vmbr1 (the Ceph public network). If using VLANs on a single physical NIC, attach one adapter to vmbr0 (untagged, VLAN 100 in our example) and the other to the VLAN bridge (vmbr1 or vmbr0.101 depending on how you set it up). This gives each VM connectivity on both subnets.
• Nested Virtualization: In each VM’s Options, enable “Nested virtualization”. (If you set CPU type to Host, recent Proxmox versions enable nested KVM automatically, but it’s good to double-check.) This allows our Proxmox-in-VM to run its own VMs or containers if needed and generally ensures full virtualization support inside.

After configuring these settings, start each VM and proceed to install Proxmox VE inside it (just as you would on a physical machine). Go through the installer: select the VM’s boot disk, set a root password, configure network interfaces (give the first NIC an IP on the cluster network, e.g. 192.168.100.11 for pve-1, etc.), and complete the installation. Once completed, each VM will reboot into Proxmox VE. You should end up with three independent Proxmox VE instances (pve-1, pve-2, pve-3) running as VMs on your host.

Networking Recap for the Lab

Let’s summarize the network plan to ensure each node is configured correctly:

• Proxmox Cluster Network (vmbr0, VLAN 100):
  – pve-1: 192.168.100.11/24
  – pve-2: 192.168.100.12/24
  – pve-3: 192.168.100.13/24
  This network is used for Proxmox management and Corosync cluster communication. All three nodes should be able to ping each other here.
• Ceph Public/Cluster Network (vmbr1, VLAN 101):
  – pve-1: 192.168.101.11/24
  – pve-2: 192.168.101.12/24
  – pve-3: 192.168.101.13/24
  This network will carry Ceph’s public traffic (OSD replication, client I/O, and monitor traffic). It’s separate from Corosync to prevent interference.

On each Proxmox VM (node), you can set these IPs via /etc/network/interfaces or the Proxmox GUI (System → Network). Ensure the Corosync ports (5404, 5405 UDP) are free of other traffic on the cluster network. Do not run other heavy services (like NFS, etc.) on vmbr0 – it’s reserved for cluster heartbeat to maintain low latency.

Creating the Proxmox Cluster (Nested Nodes Cluster)

Now that the three Proxmox VM nodes are up and networked, we can cluster them into a single Proxmox VE cluster:

• Create Cluster on pve-1: Log into the first node (pve-1) via web UI or SSH. From the web UI of pve-1, navigate to Datacenter → Cluster, and choose Create Cluster. Give the cluster a name (e.g. “lab-cluster”) and ensure the cluster network is set to the 192.168.100.0/24 network. Click Create. (If using CLI, run: pvecm create lab-cluster on pve-1). This initializes a new cluster and starts the Corosync service on pve-1, using vmbr0 network by default.
• Join pve-2 and pve-3 to Cluster: Once the cluster is created on the first node, you’ll need to join the other two nodes. In the pve-1 UI, go to Datacenter → Cluster and click Join Information to get the join command or token. On pve-2’s web UI, go to Datacenter → Cluster and choose Join Cluster. Input the IP address of pve-1 (192.168.100.11) as the master node and the secure join token from pve-1. Alternatively, via CLI on pve-2 you can run: pvecm add 192.168.100.11 (you’ll be prompted for root password of pve-1). Repeat the join process on pve-3 as well. Successful joins will synchronize configuration (/etc/pve cluster filesystem).
• Verify Cluster Quorum: After joining, you should see all three nodes listed in the Proxmox UI under Datacenter → Nodes. You can also run pvecm status on any node via SSH; it should show 3 total nodes with quorum. At this point, the three nested VMs function as a single Proxmox cluster – you can migrate VMs/CTs between them, use shared storage, etc. (Though currently no shared storage is configured. We’ll set up Ceph next for that.)

Now we have a Proxmox cluster-of-VMs running inside our host. Next, we deploy Ceph to provide distributed storage across the three nodes.

Installing Ceph on the Proxmox Cluster

Proxmox VE has tight integration with Ceph, making it straightforward to deploy a Ceph cluster on the three nodes. Proxmox VE 8.x initially includes Ceph Quincy (v17) by default, and supports newer Ceph releases as they become available. We will install Ceph and configure it to use our two networks (cluster network for internal replication, and public network for client access).

Step 1: Install Ceph Packages. In the Proxmox web interface, select any node (pve-1) and navigate to Ceph in the side menu. If Ceph is not yet installed, the UI will offer an Install Ceph option – follow the prompts to install Ceph on all cluster nodes. (This can also be done via CLI on each node with pveceph install.) This will install the Ceph daemons and management tools on pve-1, pve-2, pve-3.

Step 2: Ceph Initial Configuration. Once packages are installed, you need to create the initial Ceph config and choose networks. In the UI, go to Ceph → Configuration (or run pveceph init on the CLI). Specify the Public Network as 192.168.101.0/24 (our Ceph public subnet) and the Cluster Network as 192.168.100.0/24 (or you can leave cluster network blank to use the same network for both, but we have a separate one). Applying this will generate a /etc/pve/ceph.conf distributed to all nodes with those network settings. (On CLI, the equivalent is: pveceph init –network 192.168.100.0/24 which sets the cluster network. Then you’d manually add public network in the conf or via GUI.) Using dual networks ensures optimized performance: Ceph will prefer the dedicated cluster network for internal replication, which improves throughput and reduces latency issues.

Now the cluster knows about Ceph, but we need to deploy the Ceph MONs (monitors), MGR (manager), and OSDs (object storage daemons) to actually form a working storage cluster.

Adding Monitors and Manager daemons

Ceph monitors maintain maps of the cluster state and require quorum to operate. In a healthy Ceph cluster, you need at least 3 monitor nodes for high availability. By default, the first node (pve-1) will already have a Ceph MON after initialization. We will add monitors on pve-2 and pve-3 as well:

• Deploy MON on pve-2 and pve-3: In the Proxmox GUI on each of the other nodes, go to Ceph → Monitor and click Create (or use CLI: pveceph createmon). This will start a Ceph MON service on that node and integrate it into the cluster. After doing this on both pve-2 and pve-3, you should have three monitors total (one per node). With 3 MONs, the Ceph cluster can achieve quorum (two out of three can outvote a dead one). This is the minimum for a fault-tolerant Ceph monitor cluster.
• Ceph Manager: Ceph manager (MGR) is another small daemon that provides additional metrics and services (dashboard, auto-scaling, etc.). Proxmox will usually create one MGR on pve-1 automatically during init. You can verify under Ceph → Services that at least one manager is running. It’s optional to add a second standby manager on another node (pveceph mgr create via CLI) for redundancy, but not strictly required in a small lab cluster.

At this stage, Ceph’s core cluster services are up: monitors (mon) and manager (mgr). Ceph will be in a heartbeat state waiting for storage daemons (OSDs) to come online next.

Creating OSDs (Ceph Storage Disks)

Finally, we add OSDs – the data storage daemons – using the extra disks we attached to each VM. In Ceph, each OSD typically corresponds to one dedicated disk/SSD. We prepared (for example) a 100 GB unused virtual disk on each VM for this purpose.

Repeat the following on each node (pve-1, pve-2, pve-3) for one of the extra disks:

1. In the Proxmox web UI, select the node (e.g. pve-1), go to Ceph → OSD, and click Create OSD. In the dialog, choose the unused disk (e.g. /dev/sdb) as the Block Device. No separate DB device is needed since we’re using all-SSD and this is a small lab. Confirm to create the OSD. (The wizard will partition and format the disk for Ceph BlueStore and launch an OSD daemon on it.)
2. Alternatively, via CLI on that node you can run:

pveceph createosd /dev/sdb

(Replace /dev/sdb with the actual device name of your extra disk on that VM.)

Do this for pve-1, pve-2, and pve-3 (using their respective second disk). After creation, each node will have one OSD running. The Proxmox GUI will list 3 OSDs total, all should be Up and In the cluster. Ceph will automatically rebalance and replicate data across them.

Note: Our lab now has 3 OSDs (one per node). Ceph’s default replication factor is 3, meaning each piece of data will be stored on all three OSDs (one copy on each node). This provides redundancy—any one node (or disk) can fail and Ceph will still have 2 copies of data. In real production clusters, it’s recommended to have a larger number of OSDs (e.g. 12 or more across 3+ nodes) for better performance and fault tolerance. But for a functional lab, 3 OSDs with size=3 replication is sufficient. (You could also configure size=2 in a pinch to save space, but that’s not fully safe if one node fails.)

With monitors and OSDs in place, the Ceph cluster should reach HEALTH_OK status. You can create a Ceph pool (e.g. a replicated pool with size 3) via the Proxmox UI or CLI (ceph osd pool create). Proxmox will also allow you to add a CephFS or an RBD storage entry to actually use the Ceph storage for VM disks or container volumes.

Performance Tuning Tips for Ceph on Proxmox VE

Running Ceph in a nested, small-cluster environment can be challenging performance-wise. Here are some tips and best practices to get the most out of this setup:

• Use High-Performance Networking: As mentioned, use at least 10 GbE for your Ceph network. Ceph is network-intensive, and a 1 GbE link will become a bottleneck in a distributed storage cluster. Low latency and high bandwidth between nodes improve replication speed and client I/O. If possible, isolate Ceph traffic on its own physical NIC or VLAN to avoid contention with other traffic.
• All-SSD Storage: Small Ceph clusters perform significantly better with SSD or NVMe drives for OSDs. Fast storage media improve IOPS and reduce recovery times. In a three-node cluster, a single OSD failure means the remaining two must rebuild data – using SSDs helps complete this rebuild quickly, minimizing the window of vulnerability. Avoid using slow HDDs in labs; if you must, consider adding an SSD as a dedicated WAL/DB device for those OSDs (though that adds complexity).
• Tune Ceph for Small Cluster: Some Ceph settings can be adjusted for a low-node-count cluster. For instance, you might reduce the replication size to 2 (to save space or speed up writes) only if you can tolerate less redundancy. You can also lower heartbeat timeouts to detect failures faster in a small cluster. However, tread carefully and consult Ceph documentation before changing defaults. Generally, the defaults (size=3, min_size=2) are safest even if performance is a bit lower.
• Scale Out for Performance: Remember that Ceph’s throughput improves by adding more OSDs and nodes. More OSDs mean more disks sharing the load (more IOPS and bandwidth), and more nodes mean more CPU and network capacity in the cluster. If your hardware allows, try assigning two OSD disks per node (for 6 OSDs total) instead of one – this can nearly double the overall storage performance in the cluster. Similarly, giving each VM more CPU cores or RAM can help if you observe CPU bottlenecks (Ceph OSDs are fairly CPU-intensive). That said, Ceph in a 3-node setup will not reach the same raw performance as a single local disk or a ZFS mirror on one node; there is inherent overhead in maintaining multiple copies and network replication. Ceph prioritizes consistency and redundancy over sheer single-VM performance, so expect some latency penalty in exchange for data safety.
• Optimize Proxmox and VM Settings: Ensure that the virtIO drivers (for disk and network) are being used in the nested VMs for optimal performance. Using CPU type “host” (which we did) and enabling the IO thread setting on virtIO SCSI disks in Proxmox can improve disk performance for busy VMs. You might also want to disable any unnecessary services on the Proxmox VMs to free up resources for Ceph (for example, if you’re not using local storages on those VMs, you could disable those plugins).
• Monitoring and Benchmarks: Use tools like ceph -s to monitor cluster health and rados bench to benchmark raw Ceph throughput. Within a VM (on the nested cluster), you can run fio benchmarks against a Ceph RBD disk to gauge performance. This will help identify if the bottleneck is CPU, network, or disk. Tuning Ceph is an iterative process – for instance, you might tune the number of placement groups (PGs) for your pool if the default is not optimal (Proxmox usually auto-calculates this fairly well for small clusters).

By following these tips, you can squeeze the most performance out of your nested Ceph lab. Just keep in mind the lab’s constraints – a true production Ceph cluster with more nodes, more disks, and enterprise networking will always outperform a tiny 3-node setup. Our goal here is to get reasonable performance for testing and learning.

Backup Strategies and Data Protection

Even though Ceph provides data redundancy across nodes, it is not a backup solution on its own. Treat the Ceph-backed storage in your lab as you would any production storage when it comes to backups and recovery planning:

• Regular Backups: Implement a schedule to back up important VM data or configurations. This could mean using Proxmox’s built-in backup tool (vzdump) to dump VM images to an external disk or NAS. Ceph’s redundancy will protect against hardware failure, but it won’t prevent data corruption or accidental deletion from replicating to all copies. Only an independent backup can save you in those scenarios.
• Snapshots: Take advantage of Proxmox and Ceph’s snapshot capabilities. Proxmox can create snapshots of VMs on Ceph RBD storage almost instantly. Regular snapshots (daily/weekly) allow you to roll back if an experiment or upgrade in the lab goes wrong. However, snapshots are not a long-term backup (they reside in the same cluster), so combine them with off-cluster backups.
• Offsite or Offline Backups: If your lab data is important, consider copying backups off the main host – e.g., to an external USB drive or cloud storage. In a home lab, this might be as simple as exporting VM disks or using tools like Proxmox Backup Server (PBS) to send backups to another system. This protects against catastrophic failure (like if your one host’s PSU surge corrupts all disks) or ransomware that could affect the running cluster.
• Ceph FS vs RBD for Backup: If you set up CephFS (a distributed filesystem) on the cluster, you could also use it to store backups from within VMs or as a target for backup scripts. Just remember that CephFS is also part of the same cluster, so while it adds convenience, it doesn’t replace true external backups.

In summary, always have a backup plan even for a lab cluster. Ceph’s durability is excellent for hardware failures, but it doesn’t absolve you from maintaining backups for user errors or disasters. Practicing backup and restore procedures in your lab is also a great way to prepare for real-world production scenarios.

Use Cases for a Nested Proxmox + Ceph Lab

This kind of all-in-one Proxmox/Ceph lab is extremely useful for IT professionals, students, or enthusiasts. Here are some practical use cases and benefits of building this environment:

• Learning and Certification: If you’re studying for certifications or just learning, a nested lab lets you practice cluster operations, Ceph administration, and troubleshooting in a contained setting. You can safely experiment with commands like ceph osd out/in, network partitions, or Proxmox HA without risking production.
• Testing Upgrades and Configurations: Before rolling changes into a real cluster, test them here. For example, you can simulate a Proxmox version upgrade or a Ceph major version upgrade (e.g., Quincy to Reef) on the lab cluster to observe any issues. You can also try different Ceph pool settings, enable CephFS, or integrate Proxmox Backup Server and see how it interacts with Ceph. This lab is a sandbox for “try before you apply”.
• Performance Experimentation: Use the lab to benchmark how Ceph performs with various settings or under certain workloads. For instance, you might compare Ceph RBD performance to local storage, or test the impact of adding an extra OSD per node. While the absolute performance won’t match real hardware clusters, the relative differences and behaviors can be insightful.
• Simulating Failures: Because it’s not critical if something breaks here, you can practice failure scenarios: e.g. power off one of the Proxmox VM nodes to see how Ceph handles a node outage, then bring it back and watch data rebalancing. You can test Proxmox VM live migration with Ceph storage (since all nodes share the storage pool). This experience is invaluable for understanding high availability.
• Development and CI/CD: If you develop software that interacts with VM infrastructure or cloud storage, a Proxmox+Ceph lab can serve as a mini private cloud. For example, you could integrate it with Kubernetes or OpenStack for testing (Ceph is often used as backend storage for those). Or use the cluster to test backup tools, monitoring solutions, etc., in a realistic multi-node environment.

In essence, a nested Proxmox and Ceph lab provides a cost-effective, space-saving way to emulate a full-fledged enterprise virtualization and storage stack. It’s a playground for hands-on learning and trying out use cases that would otherwise require a lot of physical hardware. Once you’re comfortable here, you’ll be better prepared to manage or design real-world Proxmox/Ceph deployments.

Final Check and Usage

At this point, you have a nested Proxmox cluster with Ceph up and running! In the Proxmox web UI, each node should show under Ceph → OSD one OSD, and under Ceph → Monitor the three monitors with their status. The Ceph dashboard (if installed) would also show overall health. The setup can tolerate the failure of any one node or OSD (since we used triple replication). You can now create a Ceph RBD storage pool in Proxmox (Datacenter → Storage → Add → RBD) backed by this Ceph cluster, and start launching VMs or containers on highly available storage.

This nested lab is excellent for experimenting with Proxmox and Ceph features. You can practice rolling updates, simulate failures, or just get familiar with Ceph’s admin tools – all on a single machine. Despite being a lab, it’s a fully functional hyper-converged cluster: compute and storage services are all intertwined across the three virtual nodes