Container Escapes 101

AppSec Village

@ DEF CON 34

Hi, I'm Natalie!

I do application security, supply chain threat modeling, and software factory stuff for the public sector ... now with humans and AI.

I work in the public sector.

All opinions and snark my own.

Where we're going

(start some setup)
What's a container?
The security boundaries around our container
What do we mean by escape?
Where this fits in an attack
What do we have to work with here? (enumerate)
Escape time!!!
Some thoughts on AI 🤖
Where to learn more 📚

Setup!

From a vanilla Ubuntu VM (24.04 tested, both x86_64 and aarch64 acceptable)

wget https://raw.githubusercontent.com/some-natalie/some-natalie/refs/heads/main/container-security/ubuntu-setup.sh
wget https://raw.githubusercontent.com/some-natalie/some-natalie/refs/heads/main/container-security/pull-all-images.sh

sudo bash ubuntu-setup.sh
# (reboot to apply all those updates)

bash pull-all-images.sh

What's a container, anyways?

Take a trip in the wayback machine

Traditional deployment (apps on an OS on hardware), virtualized deployment (VMs on a hypervisor), container deployment (containers on a container runtime)

But why though?

Managing this took time.
Reducing "waste" needed skill.
Performance tuning wasn't easy either.
This costs time and money.

Animation of differently-sized workloads being packed into a fixed CPU-by-memory virtual machine

Constraint

Containers are an abstraction

resource management
- reduce waste
- scales where needed
"reusable bits" such as
- authentication
- load balancing
- metrics and observability
- log forwarding!
- ... so much more ...
speeds time to production

Workloads of different sizes packed into a single virtual machine's CPU and memory

Patrick Star meme: let's take all our app's complexity and move it somewhere else

I was promised simplicity

Run as many copies as needed at the same time, in parallel, with no time needed to scale up/down
Remove dependency hell across different processes that share the same host machine
Declare dependencies between services without hacky scripts or init systems
"Lift and shift" from my makefiles or Jenkinsfiles into a Dockerfile to build and run anywhere I want
Get rid of Puppet (or Ansible, Chef, etc...)
Separate persistent states (data) from ephemeral workloads (tasks)
"Livestock not pets"

every abstraction doubles your threat model, over an animated Rube Goldberg machine

What is a container, anyways?

(the ten-thousandth iteration of software distribution)

If it isn't a virtual machine, what is it?

Container stack diagram — apps and dependencies in containers, on a container runtime, on an OS, on hardware. 'You are here' points at one container.

It's a Linux process that carries along its' own dependencies.

✨ It isn't as isolated as you think ✨

Honestly, it isn't really isolated at all. At least not by default.

But it's convenient!

The same layer diagram, plus the OCI configuration: user, entrypoint, CMD, environment variables, and more

Security boundaries

✨ A container is a process. ✨

Hardware isolation is fantastic, but can be inefficient. I remember rebooting machines remotely and a few very uncomfortable minutes of waiting to know if I had to go anywhere to fix something. A virtual machine is native to the kernel of a hypervisor (read — first class citizen) for isolation. It's a long-lived thing too. A container is a process on the OS, with some special guardrails and abstractions to make developing easier than the other two options. It isn't _usually_ something you keep around forever by "powering off or on" like a VM. Notice the engineering trade-off there? We lost some isolation from a VM. We lowered overhead for better resource utilization — both people and time. We are not losing all isolation, though, especially if we're mindful of what a container is and how it works.

Secure Computing (seccomp)

The foundation we build on!

Want to read a file? open() and read()
Write to the network? socket() + friends
Make a new directory? mkdir()

It's all system calls (syscalls). There's hundreds. makelinux.github.io/kernel/map

Do you need hundreds?

seccomp is like an electric fence.

Jurassic Park — animation of a boy climbing an electric fence, thrown backwards when it energizes

I didn't want to become a kernel expert today

Let's chart our escapes based on what a container is under the hood.

namespaces - what a container can SEE
cgroups - what a container can HAVE
capabilities - what a container can DO
overlayfs - the weird filesystem that stacks on top of itself

AppArmor or SELinux are part of the host's kernel called Mandatory Access Control. They make sure each process can only touch things it's allowed to.

Namespaces (see)

How single resources, like a network card, are shared across many processes without those processes seeing each other.

Hostname (uts)
System time (time)
Inter-process comms (ipc)
Control groups! (cgroup)
Mount points (mount)
Network (network)
Process IDs (process)
User IDs (user)

Minimizing what a container can see

Reduces attack surface
Reduces amount of information exposed for living off the land

docker run -it --uts=host ubuntu:jammy bash gives a shell whose hostname is the host's; without --uts=host the hostname is the container ID

Control groups (have)

It's a weird filesystem.
Defines and sets boundaries on what a process can have.
Protect it. Messing with it can be part of a denial of service attack.

cat /sys/fs/cgroup/cpu.stat and memory.pressure showing CPU usage and memory pressure counters

ls -lah /sys/fs/cgroup showing cgroup.controllers, cpu.stat, memory.pressure, io.stat, system.slice, user.slice and more

Capabilities (do) - let's get dangerous!

Used to be root/admin/everything or nothing at all
Now there's about 40 unique permissions
CAP_NET_BIND_SERVICE
CAP_AUDIT_READ
CAP_SYS_PTRACE
CAP_SYS_ADMIN is effectively everything

Be careful out there!

This gives us a better escape and more of a foothold once we're out.

xkcd 149: 'make me a sandwich' / 'what? make it yourself' / 'sudo make me a sandwich' / 'okay'

Mandatory Access Controls

One more line of defense - SELinux or AppArmor
Part of the host - each process (container) can only touch/do things based on
- User
- Role
- File locations, processes, etc - CONTEXT
SELinux is more RedHat
AppArmor is more Debian

😩 The reason I bring this up is that it's common to disable these! It's usually the first suggestion and it's easier to leave it off than troubleshoot and write a new policy.

Fun police are here to stop more shenanigans. The most common MAC systems are AppArmor or SELinux. These act as watchdogs to ensure each process (container or not) is only touching resources it's allowed to based on the user, their role, and the files/processes/tasks attempted. The reason I bring this up is that it's common to disable these. It's often the top-rated answer on StackOverflow or the first "fix" in a blog post that ranks high in search results. It is always a bad idea to disable these. I asked a popular AI coding assistant how to fix an error where AppArmor stopped me from doing something and copy/pasted in the error. It told me how to disable AppArmor and then warned me at the very end that that may have security implications. So naturally, we'll be disabling these for our demo!

What's an escape?

Container escape — an arrow from an app inside a container down to the host operating system, past the container runtime

Types of escapes

Kernel vulnerabilities ("dirty pipe" or CVE-2022-0847)
Runtime vulnerabilities (2024's "leaky vessels" CVEs for runc and BuildKit)
Insecure configuration at runtime

We're focusing on that third one.

Most common (in my experience)
Least "alarmed on" and most likely to get dismissed (in my experience)
... also most reliable in a workshop ...

Where this fits in an attack

Several hosts each running containers on a runtime, all under a container orchestration layer (usually Kubernetes), with an image registry alongside

Enumeration

Who am I? - Let's get oriented and figure out what we have in our container.
Our shared kernel - Containers are processes that share a kernel. What can we see?
Are we capable? - What sort of capabilities do we have?
Seccomp is your friend - Seccomp filters what a container can do. Let's learn what's been set for us.
Are we in a microVM? - With microVM runtimes gaining popularity, how do you know if you're in a container?

Escapes!!

Persistent storage - Persisting through persistent storage
Chrooooooooot - chroot right on out of a container's filesystem.
Runtime groups - The default "docker" group is basically root.
Shared runtime sockets - Use a shared socket to create a new container
Messing with the host's memory - Let's modify a process in memory to do fun things! (example is x86_64 only)
Vulnerable apps in the wild - You're typically not SSH'd into the container host, so ... how does this still work in the wild?
Finding secrets in image layers - Finding secrets in a container image's layers can be done at runtime if you have access to download it and a shell or access to the host's socket or access to the registry.

Workshops

Two parts
- Enumeration
- Escapes!!
Need a screen reader?
Need to make fonts bigger?
Want to work at your own pace?
Revisit things later?

some-natalie.dev/container-escapes-workshop

(slides, notes, workshop examples, etc.)