Building a custom root filesystem for Firecracker

In the previous article, we explored getting started with Firecracker. In that post, we used the root filesystem that’s provided by their CI, but in practice we’d likely want to build our own. This will cover building our own root filesystem with docker that we expand as we continue to dive into firecracker

Table of contents

Installing docker

We build the rootfs with docker, so it needs to be installed and running. The host we provisioned in the previous post is Amazon Linux 2023, so:

sudo dnf install -y docker
sudo systemctl enable --now docker

# Let your user talk to the docker socket without sudo, then re-evaluate
# group membership in the current shell.
sudo usermod -aG docker $USER && newgrp docker

If you’re on a different distro, follow docker’s install instructions and make sure your user is in the docker group. The group step matters: the build script below calls docker directly, so without it every docker command fails with a permission denied ... /var/run/docker.sock error.

The gist

What we want to do is to take a really lightweight docker image (like alpine) and essentially mount / onto our host filesystem. We can then mount an EXT4 file onto our filesystem & copy the necessary directories into it, creating our rootfs image.

Let’s start by defining our Dockerfile

# ./Dockerfile
FROM alpine

WORKDIR /root

# Set root password
RUN echo "root:root" | chpasswd

# Download dependencies
RUN apk add --update --no-cache \
    openrc \
    util-linux

# Setup login terminal on ttyS0
RUN ln -s agetty /etc/init.d/agetty.ttyS0 \
    && echo ttyS0 > /etc/securetty \
    && rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot
RUN rc-update add devfs boot \
    && rc-update add procfs boot \
    && rc-update add sysfs boot \
    && rc-update add local default


# The /root directory will contain a script that copies
# files from the mounted docker volume into the mounted
# EXT4 file
COPY root /root

and the script that copies from the Docker volume into the mounted EXT4 file

# ./root/copy-to-rootfs
ROOT_DIR=$1
if [ -z $ROOT_DIR ]
then
    echo "Missing argument: need a root directory"
    exit 1
fi
echo "Copying rootfs directories to $ROOT_DIR"
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C $ROOT_DIR; done

# The above command may trigger the following message:
# tar: Removing leading "/" from member names
# However, this is just a warning, so you should be able to
# proceed with the setup process.

for dir in dev proc run sys var; do mkdir $ROOT_DIR/${dir}; done

# Delete this file from the mnt directory
rm $ROOT_DIR/root/copy-to-rootfs

and then finally the build script that puts everything together.

set -eu
ROOTFS_FILE=$1
DOCKER_DIRECTORY=$(dirname "$0")
ROOTFS_DIR=/tmp-rootfs
HOST_ROOTFS_DIR=/tmp$ROOTFS_DIR
DOCKER_TAG='rootfs-builder'


# Cleanup from previous attempt
rm -f $ROOTFS_FILE
sudo umount $HOST_ROOTFS_DIR || true
sudo rm -rf $HOST_ROOTFS_DIR

# Create file. You may need to increase the count or blocksize
# if your rootfs get's too big
dd if=/dev/zero of=$ROOTFS_FILE bs=1M count=1024

# Create empty filesystem
sudo mkfs.ext4 $ROOTFS_FILE

# Make sure directory is created
mkdir -p $HOST_ROOTFS_DIR

# Mount the filesystem
sudo mount $ROOTFS_FILE $HOST_ROOTFS_DIR


# Build our custom rootfs builder
docker build --tag $DOCKER_TAG $DOCKER_DIRECTORY

# Runs the container, mounting a volume
docker run --rm -v $HOST_ROOTFS_DIR:$ROOTFS_DIR \
    $DOCKER_TAG sh copy-to-rootfs $ROOTFS_DIR

sudo umount $HOST_ROOTFS_DIR

Cool, we can build our rootfs now! Save the script above as build-rootfs.sh and run it (without sudo — it elevates only the individual mount/mkfs steps that need it). Running the whole script as root would leave /tmp/rootfs.ext4 owned by root, and Firecracker — which opens the drive read-write — would then fail with a PermissionDenied error on boot.

bash build-rootfs.sh /tmp/rootfs.ext4

Let’s place it at /tmp/rootfs.ext4 and start our new VM!

cat << EOF > /tmp/vmconfig.json
{
  "boot-source": {
    "kernel_image_path": "/tmp/kernel.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "is_root_device": true,
      "is_read_only": false,
      "path_on_host": "/tmp/rootfs.ext4"
    }
  ]
}
EOF

rm -f /tmp/firecracker.socket \
&& /tmp/firecracker \
--api-sock /tmp/firecracker.socket \
--config-file /tmp/vmconfig.json


# Login with root:root

Initd services

Now we have full control over the rootfs & we can start doing some cool stuff. For example, we can pass in parameters via the kernel boot args & do something based on those parameters during the boot phase by creating a initd service. Let’s log out the kernel args for now

#!/sbin/openrc-run
# ./etc/init.d/boot-arg-logger

start() {
    einfo "starting kernel boot argument logger"

    eindent

    BOOT_ARGS=$(cat /proc/cmdline)
    einfo "Kernel boot args were: $BOOT_ARGS"

    eoutdent


    return 0
}

stop() {
    einfo 'stopping kernel boot argument logger service…'
    return 0
}

mark this file as executable via chmod +x and we’ll update our docker file to copy in the ./etc folder + add the boot-arg-logger service. Note that rc-update add boot-arg-logger with no runlevel argument adds the service to the sysinit runlevel, so it fires early in boot.

# ./Dockerfile
...
# The /root directory will contain a script that copies
# files from the mounted docker volume into the mounted
# EXT4 file
COPY root /root

COPY etc /etc

RUN rc-update add boot-arg-logger

Now when you start firecracker you’ll see the following lines

OpenRC 0.52.1 is starting up Linux 5.10.209 (x86_64)
...
* starting kernel boot argument logger
*   Kernel boot args were: console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5

You can view code samples for this article on my github .