The Fleet: Six Machines, and What the Hypervisor Can't Do

The lab was supposed to be one machine. Part 1 made the consolidation case — one Proxmox box, twenty containers, everything snapshotted and segmented. Then I built a performance-sampling tool, asked the obvious question — does it behave the same on ARM? — and discovered that the machine I had consolidated everything onto couldn’t answer it.

This post is about the fleet that grew around that lesson. Six machines, and none of them is here for fun: each one exists because I hit a problem the hypervisor could not solve, and each one taught me something I wouldn’t have learned from a VM. That’s the whole point of this lab — it’s a classroom that happens to run services.

The center, and one cheap trick worth stealing

The Proxmox host from part 1 is still the center of gravity: an Ultra 9 285K, 96 GB of DDR5-6600, and three 2 TB Gen4 NVMe drives carrying separate ZFS pools, plus an 8 TB mirrored pool on spinning disks for bulk data.

The HDD pool taught me the first lesson. Spinning disks stream large files fine, but metadata is where they die — walking a directory tree of a few hundred thousand files means a seek per lookup, and ls starts to feel like an I/O benchmark. The fix was a ZFS special vdev: a mirrored pair of 256 GB SATA SSDs attached to the HDD pool that holds its metadata (and optionally small blocks). Directory walks hit flash; big sequential reads still stream off the platters. Two leftover SSDs turned out to be the cheapest upgrade a spinning pool can get. One caveat that matters: the special vdev is the pool — lose it and the pool is gone — which is why it’s a mirror and not a single drive.

The counters that wouldn’t count

The perf tool is where consolidation broke. Sampling profilers lean on the PMU — the CPU’s performance monitoring unit, the hardware counters behind perf stat’s cycles and cache-misses. And a hypervisor is exactly the wrong place to learn how PMUs behave: virtualized counters are incomplete when they’re exposed at all, the hypervisor multiplexes them behind your back, and the emulated ARM VM from part 1 has no real PMU whatsoever — QEMU proves your code runs on ARM, not how it performs.

So the tool got tested where counters tell the truth — on bare metal, across three genuinely different microarchitectures:

i5-3330 — Ivy Bridge, Intel’s PMU, 2012 vintage. An old baseline is a feature: if the tool works here, it isn’t accidentally depending on anything modern.
A10-9600P — AMD’s PMU, which is not Intel’s PMU. Same perf API, different counters, different quirks. You don’t learn that from documentation; you learn it when the same code returns different shapes of data.
Snapdragon 845 — the ARM PMU, big.LITTLE and all, in a phone (more on that below).

That test matrix is the lab’s thesis in miniature: the hypervisor virtualizes CPUs well and performance counters badly. Anything that touches hardware this directly needs real silicon.

A phone is the cheapest ARM dev board

The ARM machine in that matrix is a Poco F1 — Snapdragon 845, 8 GB of RAM, 256 GB of storage — running a NetHunter kernel. NetHunter is security tooling, but that’s not why it’s here: the point is a rooted device with a replaceable kernel. That combination turns a retired phone into an ARM development board with better specs than most actual dev boards — four Cortex-A75s and four A55s of real big.LITTLE scheduling, a real ARM PMU, and a battery-backed power supply built in.

Every “works on my machine” assumption a career of x86 quietly builds up gets audited the first time your tool runs on this thing.

The bench that has real ports

Firmware work was the second thing the hypervisor lost. Flashing embedded boards — like the ESP32 wake gateway — means serial consoles and USB devices that reset and re-enumerate mid-operation. USB passthrough into a VM can be forced, but a device that drops and reattaches during a flash is a great way to end up with half-written firmware, and the failure is always ambiguous: the board, the cable, or the passthrough?

The answer is a machine nobody would miss: an i5-3330 desktop with 16 GB and an assortment of leftover disks. Real USB controllers, a real serial port, no virtualization layer to suspect. When a flash fails there, the board is actually broken — and removing a layer of doubt is worth more than the electricity it burns. It doubles as the Ivy Bridge test target above.

The machines that run the machines

The rest of the fleet is about operating the lab rather than being experimented on. An i5-1240P laptop — 32 GB, a 2 TB and a 512 GB Gen4 NVMe, Windows 11 — is the daily driver and control plane: every console, dashboard, and SSH session in the lab happens from here. And a MacBook Air M4 with 24 GB is the portable half — on the Tailscale mesh from part 1, so “at the lab” is a network property, not a physical one. The A10 laptop rounds things out as the sacrificial box: risky OS installs and one-off servers land there first, where a bad day costs nothing.

The deliberate redundancy: the control plane runs Windows, the portable runs macOS on ARM, the lab runs Linux. Three operating systems and two ISAs in daily rotation means cross-platform assumptions get caught by breakfast.

Takeaways

Consolidation is right up until the work touches hardware. PMUs, serial ports, USB quirks — that’s where bare metal earns its keep.
A hypervisor virtualizes CPUs well and performance counters badly. Test anything perf-shaped on real silicon, across more than one vendor’s PMU.
A rooted phone with a replaceable kernel is the cheapest real-ARM test device you can own — better specs than a dev board, battery included.
A mirrored special vdev is the cheapest upgrade a ZFS spinning pool can get: metadata on flash, bulk data on platters. Mirror it — it’s pool-critical.
Keep one machine nobody would miss. Risky experiments need a home where failure is free.
Next in this series: the strangest tenant on the always-on host — the AI coworker in an LXC container.