VMs vs LXC vs Docker

When should I use a VM vs LXC vs Docker?

Virtual machines (VMs), Linux containers (LXC), and OCI containers (Docker) are all virtualization technologies that allow us to separate programs from the host system. This provides added security and a certain degree of "oops" protection when running untrusted or experimental code.

For some time, I've struggled to figure out how I should decide which tool is appropriate for which situation. Just the other day I had an epiphany and realized this simple decision process:

The TL;DR is:

1. If I need to run a separate CPU architecture or separate OS kernel, I use a VM.
2. If I don't care about running a separate kernel but still need a full init system, I use LXC.
3. If I just want to run an isolated application, I use OCI containers.

Overview of Each Tool

Virtual Machines

A virtual machine is a full system also called a "guest", that usually includes a bootloader, kernel, initramfs, and root filesystem. They run on virtualization environments (VEs) like qemu, xen, and virtualbox. The VE or host provides the guest kernel with virtual hardware, i.e. code that acts like hardware and provides a host-controlled bridge between the real hardware and guest. This makes the guest VMs highly isolated from the host, and the guest can even run kernels that are compiled for different architectures than the host—for example a RISCV guest running on an x86_64 host.

PROS

VMs are the most flexible option, as the host VE has complete programmatic control over the virtual hardware the guest has access to. VMs can also be easily "broken out" into a bare-metal machine by writing their backing storage to a physical storage device and putting it in a separate machine.

CONS

VMs require an entirely separate kernel and thus take up a lot of memory. Because their hardware is fully virtualized, there is also additional computational overhead needed to manage access to the virtual hardware.

LXC

LXC is the name given to a bundle of userspace tools that leverage Linux kernel containment features like chroot(1) and cgroups(7) to create environments that are separate from the host but without requiring full hardware virtualization like VMs. If VMs virtualize hardware resources, LXC containers virtualize kernel resources.

LXC containers are primarily intended to run a full OS (minus the kernel), so they usually include an init system like systemd(1) or openrc. This makes them an excellent drop-in replacement for VMs when kernel separation is not a requirement.

PROS

Full OS virtualization without needing to run a separate kernel.

CONS

Less flexibility than VMs due to reliance on the same kernel and slower startup than OCI containers due to the init system startup time.

OCI Containers (Docker)

OCI containers—formerly known as Docker containers—rely on the same Linux kernel containment features as LXC, but were built with a different use-case in mind: microservices.

Each OCI container usually only runs a single process and therefore does not need any sort of init system. Additionally each container is meant to be run from a stateless base filesystem for increased reproducibility, so if a containerized application malfunctions then restarting the container will run it from a fresh state.

PROS

Fast startup and reproducibility.

CONS

Only intended to run a single microservice-oriented application.

Summary

Any of the tools mentioned above can be used to run any application. The question about which tool to use comes down to resource consumption and efficiency.

VMs are resource-intensive but provide the greatest flexibility.

LXC containers are less resource-intensive but are still meant to run full systems.

OCI containers are the least resource-intensive but require the adoption of a microservice-oriented application deployment strategy.