LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Booting the NetBSD 11 Kernel in 10ms?!? Can You Really Do That? Let's Put That Project's Brag to the Test!

NetBSDI’ve never been a big NetBSD guy.  I played with it a long time ago (like 30 years ago) when I had a 68K Mac and wanted to run a Unix-like OS on it, and NetBSD was the only option.  Since then I’ve fooled around with it once or twice, but I just never had a need that OpenBSD or Linux didn’t meet better.

But recently I saw that NetBSD 11 was released, and it included this brag:

New MICROVM kernel for x86, supporting both i386 and amd64, NetBSD 11.0 introduces a dedicated MICROVM kernel designed for extremely fast virtual machine boot, leveraging PVH boot, VirtIO MMIO, and multiple kernel optimizations, it can boot in about 10 ms on 2020-era x86 CPUs.

That certainly sounds cool!  My home Proxmox runs on an i5-8250U which is a little earlier than that.  It was released in 2017 Q3.  So maybe it won’t be quite 10ms, but it should still be pretty fast.

Why would you care?  For places like my home lab, you probably won’t.  Whether it’s 10ms or 10 seconds, it doesn’t make much difference.  But if you were running on virtual fabric where you need fast ballooning, very quick failover, or the ability to restart quickly, that kind of capability would be invaluable.

What is MICROVM?

A MICROVM is a virtual machine built around the idea of presenting the guest with the smallest possible set of virtual hardware.  That means that there’s a lot less hardware discovery and initialization at startup.

A normal Proxmox VM emulates something that looks a lot like a physical PC: BIOS or UEFI, PCI/PCIe buses, ACPI, graphics card, SATA/SCSI/IDE controllers, network adapters, various legacy devices, etc.

A MICROVM strips most of that away. Instead, it typically has just CPU, RAM, a serial console, VirtIO disk, and VirtIO network. With QEMU’s microvm machine type, there’s normally no PCI bus and no ACPI, and devices are exposed through VirtIO-MMIO instead of VirtIO-PCI.

The main benefit is less virtual hardware to initialize and emulate. One subtle point: a MICROVM is still a real hardware-virtualized VM. It is not a container. NetBSD still boots its own kernel and has its own memory, processes, filesystem, and kernel isolation boundary.

Let’s Try It!

I went down the path of creating a NetBSD VM in the Proxmox GUI, but that was a dead end.  I naively thought you’d just swap out the kernel and you’d golden.  But MICROVM needs help from the QEMU host.

The Proxmox host’s QEMU supports microvm, but Proxmox’s VM-management layer still assumes a more conventional PC-style machine layout. In particular, Proxmox automatically injects devices such as PCI bridges and expects its normal disk/NIC models, while QEMU microvm deliberately has no PCI bus.  There’s some discussion here on the Proxmox forums about lack of support, and if you’re adventurous there is a patch someone created, but…my Proxmox is “production” in the sense that various important (to me) things run on it, so I’d rather not.

But let’s give it a try at the command line with qemu.  The NetBSD web site has instructions, but you’ll need a couple tweaks.

First, that link to the live image is dead.  Here is an updated link.

However, that live image doesn’t use MBR but rather GPT.  So you’ll get this if you use the recipe on NetBSD’s MICROVM page.

[   1.0000000] WARNING: system needs entropy for security; see entropy(7)
[   1.0000000] entropy: ready
[   1.0000000] [ Kernel symbol table missing! ]
[   1.0000000] NetBSD 11.0 (MICROVM)           Notice: this software is protected by copyright
[   1.0000000] Detecting hardware... (QBOOT    000000000000)
[   1.0097601]  done.
[   1.0097601] vfs_mountroot: can't open root device
[   1.0097601] cannot mount root, error = 16
[   1.0097601] root device (default ld0a):

At that point, I entered a ? and it gave a list of potential disk candidates.  I don’t know why I remembered that trick of yore…must be some 30-year-old brain cell that didn’t expire when it was supposed to.  Well you know what they say, cache expiry is the hardest problem in computer science.

[   1.0097601] root device (default ld0a): ?
[   2.1342822] use one of: dk0 dk1 ld0[a-p] vioif0 wedge:EFI system wedge:NBImgRoot halt reboot

wedge:NBImgRoot is what you want.

So my command was (using the same variables shown in the NetBSD example to point to the kernel and live image path):

qemu-system-x86_64 -M microvm,rtc=on,acpi=off,pic=off,accel=kvm \
    -cpu host,+invtsc -kernel ${KERNEL} -append "root=wedge:NBImgRoot console=com rw -z" \
    -device virtio-blk-device,drive=hd0 -drive file=${NBIMG},format=raw,id=hd0 \
    -device virtio-net-device,netdev=net0 -netdev user,id=net0,ipv6=off,hostfwd=::2322-:22 \
    -global virtio-mmio.force-legacy=false -display none -serial stdio

And as the NetBSD site says, behold the magic:

[   1.0000000] WARNING: system needs entropy for security; see entropy(7)
[   1.0000000] entropy: ready
[   1.0000000] [ Kernel symbol table missing! ]
[   1.0000000] NetBSD 11.0 (MICROVM)           Notice: this software is protected by copyright
[   1.0000000] Detecting hardware... (QBOOT    000000000000)
[   1.0124780]  done.
[   1.0164205] kernel boot time: 29ms
Not resizing / (NAME=NBImgRoot): already correct size
The following components reported failures:
    /etc/rc.d/resize_gpt
See /var/run/rc.log for more information.
Sat Aug 15 02:02:01 UTC 2026

NetBSD/amd64 (netbsd) (constty)

login: 

Not bad!  29ms.  Ran it a few times and it was consistently between 28ms and 31ms, with 29ms the most common time.

Of course, the actual time to get all service started, etc. was slightly longer, and if this was in real production they’d be even longer – to start Postgres or whatever, for example.  But they kernel is really no longer a factor.

Pro Tip: if something doesn’t work, change the -z append= line to -v.  -z means “silent mode” and -v means “verbose mode”.”

 

No Comments

    Leave a Reply

    Some notes on commenting on LowEndBox:

    • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
    • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
    • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

    Your email address will not be published. Required fields are marked *