Building a custom root filesystem for Firecracker

In the previous article, we explored getting started with Firecracker. In that post, we used the root filesystem that’s provided by their CI, but in practice we’d likely want to build our own. This will cover building our own root filesystem with docker that we expand as we continue to dive into firecracker

Table of contents

Downloading docker

I’m going to assume you have docker installed, if you don’t you can find install instructions here on the docker website https://docs.docker.com/engine/install/debian/

The gist

What we want to do is to take a really lightweight docker image (like alpine) and essentially mount / onto our host filesystem. We can then mount an EXT4 file onto our filesystem & copy the necessary directories into it, creating our rootfs image.

Let’s start by defining our Dockerfile

# ./Dockerfile
FROM alpine

WORKDIR /root

# Set root password
RUN echo "root:root" | chpasswd

# Download dependencies
RUN apk add --update --no-cache \
    openrc \
    util-linux

# Setup login terminal on ttyS0
RUN ln -s agetty /etc/init.d/agetty.ttyS0 \
    && echo ttyS0 > /etc/securetty \
    && rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot
RUN rc-update add devfs boot \
    && rc-update add procfs boot \
    && rc-update add sysfs boot \
    && rc-update add local default


# The /root directory will contain a script that copies
# files from the mounted docker volume into the mounted
# EXT4 file
COPY root /root

and the script that copies from the Docker volume into the mounted EXT4 file

# ./root/copy-to-rootfs
ROOT_DIR=$1
if [ -z $ROOT_DIR ]
then
    echo "Missing argument: need a root directory"
    exit 1
fi
echo "Copying rootfs directories to $ROOT_DIR"
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C $ROOT_DIR; done

# The above command may trigger the following message:
# tar: Removing leading "/" from member names
# However, this is just a warning, so you should be able to
# proceed with the setup process.

for dir in dev proc run sys var; do mkdir $ROOT_DIR/${dir}; done

# Delete this file from the mnt directory
rm $ROOT_DIR/root/copy-to-rootfs

and then finally the build script that puts everything together.

set -eu
ROOTFS_FILE=$1
DOCKER_DIRECTORY=$(dirname "$0")
ROOTFS_DIR=/tmp-rootfs
HOST_ROOTFS_DIR=/tmp$ROOTFS_DIR
DOCKER_TAG='rootfs-builder'


# Cleanup from previous attempt
rm -f $ROOTFS_FILE
sudo umount $HOST_ROOTFS_DIR || true
sudo rm -rf $HOST_ROOTFS_DIR

# Create file. You may need to increase the count or blocksize
# if your rootfs get's too big
dd if=/dev/zero of=$ROOTFS_FILE bs=1M count=1024

# Create empty filesystem
sudo mkfs.ext4 $ROOTFS_FILE

# Make sure directory is created
mkdir -p $HOST_ROOTFS_DIR

# Mount the filesystem
sudo mount $ROOTFS_FILE $HOST_ROOTFS_DIR


# Build our custom rootfs builder
docker build --tag $DOCKER_TAG $DOCKER_DIRECTORY

# Runs the container, mounting a volume
docker run -it --rm -v $HOST_ROOTFS_DIR:$ROOTFS_DIR \
    $DOCKER_TAG sh copy-to-rootfs $ROOTFS_DIR

sudo umount $HOST_ROOTFS_DIR

Cool, we can build our rootfs now! Let’s place it at /tmp/rootfs.ext4 and start our new VM!

cat << EOF > /tmp/vmconfig.json
{
  "boot-source": {
    "kernel_image_path": "/tmp/kernel.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "is_root_device": true,
      "is_read_only": false,
      "path_on_host": "/tmp/rootfs.ext4"
    }
  ]
}
EOF

rm -f /tmp/firecracker.socket \
&& firecracker \
--api-sock /tmp/firecracker.socket \
--config-file /tmp/vmconfig.json


# Login with root:root

Initd services

Now we have full control over the rootfs & we can start doing some cool stuff. For example, we can pass in parameters via the kernel boot args & do something based on those parameters during the boot phase by creating a initd service. Let’s log out the kernel args for now

# ./etc/init.d/boot-arg-logger
#!/sbin/openrc-run

start() {
    einfo "starting kernel boot argument logger"

    eindent

    BOOT_ARGS=$(cat /proc/cmdline)
    einfo "Kernel boot args were: $BOOT_ARGS"

    eoutdent


    return 0
}

stop() {
    einfo 'stopping kernel boot argument logger service…'
    return 0
}

mark this file as executable via chmod +x and we’ll update our docker file to copy in the ./etc folder + add the boot-arg-logger service to the default runlevel.

# ./Dockerfile
...
# The /root directory will contain a script that copies
# files from the mounted docker volume into the mounted
# EXT4 file
COPY root /root

COPY etc /etc

RUN rc-update add boot-arg-logger

Now when you start firecracker you’ll see the following lines

OpenRC 0.52.1 is starting up Linux 5.10.209 (x86_64)
...
* starting kernel boot argument logger
*   Kernel boot args were: console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5

You can view code samples from this article on my github