Files in an in-memory /tmp count against a container's memory limit

Whisk. Published 5 October 2026. Measurements taken 4 October 2026.

Abstract

Many container platforms and some Linux distributions mount /tmp as tmpfs, a file system held in memory. We measured what happens when a program in a memory-limited Linux container writes temporary files to such a mount. In a container limited to 200 MiB, writes of up to 190 MiB to a 1 GiB tmpfs succeeded, and every write of 250 MiB or more was stopped by the kernel's out-of-memory killer at 197 to 198 MiB, while df still reported 826 MiB free on the mount. The same writes to the container's disk-backed file system completed in all runs. A file left in tmpfs kept its memory charged to the container after the program that wrote it had exited, and a second program that needed 120 MiB of ordinary memory was then killed in five of five runs. The free space a tmpfs mount reports is therefore not a reliable guide to how much can be written to it inside a memory-limited container.

1. Introduction

Temporary files are usually treated as a disk concern. A program that needs scratch space writes to /tmp and expects the only limit to be the space left on that file system. On many current systems that expectation is wrong. Docker can mount any directory as tmpfs [1], Kubernetes offers memory-backed emptyDir volumes [2], Google Cloud Run gives each container an in-memory file system [3], and Debian 13 stores /tmp in a tmpfs by default [4]. In all of these, a file in /tmp occupies memory, not disk.

The documentation of Docker, Kubernetes and Cloud Run states that this memory counts against the container's limit [1][2][3]. What the documentation does not show is how the failure presents to the person running the program. This paper measures it. We wanted to know three things: at what point a write fails, what the failure looks like from inside the container, and whether the memory is released when the writing program ends.

The question arose in practice. A business application hosted on Whisk, which processes call recordings, was repeatedly killed while it loaded a 144 MB file into /tmp. Its own memory use looked modest and the mount showed free space, so the cause was not obvious.

2. Background

tmpfs is a Linux file system whose contents live in the kernel's page cache and swap rather than on a block device [5]. Its size option caps how much the mount may hold, and the default cap is half of the machine's physical memory [5].

Containers on Linux are limited with control groups (cgroups). The cgroup memory controller tracks "page cache and anonymous memory" used by the processes in a group [6], and its statistics list tmpfs pages separately under shmem, described as "cached filesystem data that is swap-backed, such as tmpfs, shm segments, shared anonymous mmap()s" [6]. The same documentation states that "a memory area is charged to the cgroup which instantiated it and stays charged to the cgroup until the area is released" [6]. For a tmpfs file, the area is released when the file is deleted, not when the program that wrote it exits.

Two limits therefore apply to every write to a tmpfs mount inside a container: the mount's own size, and the container's memory limit. Tools such as df report only the first.

3. Method

All tests ran on Linux kernel 6.18 with Docker Engine 29.6.2 using cgroup version 1, in containers from the Alpine Linux 3.20.10 image. Each container was started with a memory limit of 200 MiB and swap disabled for the container (--memory=200m --memory-swap=200m).

Write test. A shell script (Appendix A) wrote a file of a chosen size with dd in 1 MiB blocks, then recorded the exit status of dd, the size of the file actually written, the container's memory use and shmem figure from the cgroup, and the mount's usage as reported by df. We ran five conditions against a 1 GiB tmpfs mount (--tmpfs /scratch:size=1g) with file sizes of 100, 150, 190, 250 and 400 MiB. As controls we wrote 400 MiB to the container's ordinary disk-backed file system, and 100 MiB to Docker's default /dev/shm.

Persistence test. A second script (Appendix B) ran two separate programs one after the other. The first wrote 150 MiB to the tmpfs mount and exited. The second needed a single 120 MiB buffer of ordinary memory. Together the two exceed 200 MiB, but the first program had already ended before the second began. The script then deleted the file and ran the second program again. As a control, the same sequence ran with the file on disk.

Every condition was repeated five times, each in a fresh container.

4. Results

4.1 Writes to tmpfs

In every run, the tmpfs pages were charged to the container. The cgroup's shmem figure equalled the size of the file to within 1 MiB, and total memory use rose by the same amount.

Every write above the limit ended the same way: dd received SIGKILL from the out-of-memory killer and exited with status 137 (128 + 9). It printed no error message about space, because the file system was not full. In 7 of the 10 runs above the limit, the out-of-memory killer also ended the container's main process, so the whole container exited with status 137. In the other 3, the main process survived and only the writing program was lost.

4.2 Controls

Writing 400 MiB to the disk-backed file system completed in 5 of 5 runs, in the same 200 MiB container. Memory use rose to 198 MiB during the write, because written data passes through the page cache, but that cache was reclaimed as needed and no process was killed.

Docker's default /dev/shm is a separate 64 MiB tmpfs. A 100 MiB write to it failed in 5 of 5 runs with "No space left on device" at 64 MiB, the mount's own size limit, and no process was killed. Here the mount's limit was reached before the memory limit, so the failure looked like an ordinary full disk.

4.3 Persistence after the writing program exits

A 150 MiB file left in tmpfs stayed charged to the container after its writer had exited, and a later, unrelated program was killed for want of memory. The same file on disk was also cached in memory, but the kernel reclaimed that cache to make room, and the later program ran.

5. Discussion

The results confirm the documented behaviour [1][2][3] and add three practical points.

First, the failure gives no clue to its cause. The program does not see "disk full". It is killed, and df shows the mount well below its size. In our tests df reported 826 MiB free at the moment of every failure. Anyone diagnosing the fault from inside the container would see a program that died for no visible reason.

Second, the effective size of a tmpfs mount in a container is the smaller of its size option and the memory the container has left after everything else it is doing. Setting a large size does not provide more room. Docker's documentation makes the same point: "setting a large tmpfs-size (or host-mounted tmpfs) does not give the container extra RAM outside that limit" [1].

Third, the cost outlives the program. A file forgotten in tmpfs reduces the memory left for every later program in the same container until someone deletes it. This matters most for long-running containers that run many short jobs, such as scheduled tasks, where one job's leftover files can cause a later job to fail.

For applications that handle large files, the practical consequences are to stream data rather than read it whole into a temporary file, to write large temporary files to disk-backed storage or object storage, and to delete temporary files as soon as they are no longer needed. On a platform where /tmp is in memory, the memory limit should be set to cover the application's peak memory use plus the largest temporary files it keeps.

6. Limitations

We tested cgroup version 1 only. The cgroup version 2 documentation describes the same charging of tmpfs pages [6], but we did not measure it. We used one kernel version, one container runtime (runc, through Docker) and one container image. Sandboxed runtimes such as gVisor keep their own in-memory file systems and may behave differently. Swap was disabled for the container, so tmpfs pages could not be moved to swap; with swap available, the limit would be reached later. Each condition was repeated five times. The results were the same each time except for the size written before the kill, which varied by 1 MiB, and for which process the out-of-memory killer chose.

7. Conclusion

Inside a memory-limited Linux container, a file in a tmpfs mount uses the container's memory, not disk. Writes stop when the container reaches its memory limit, whatever free space the mount reports, and the writing program is killed rather than given an error. Files left in tmpfs keep their memory charged after their writer exits and can cause later programs to fail. When /tmp may be memory-backed, large temporary files belong on disk-backed storage, and the container's memory limit has to allow for any files kept in memory.

For related measurements, see our papers on where the MP4 index sits and how that affects playback and on how late visitor-triggered scheduled tasks run.

References

1. Docker. "tmpfs mounts." Docker documentation. https://docs.docker.com/engine/storage/tmpfs/ (accessed 4 October 2026).
2. The Kubernetes Authors. "Volumes: emptyDir." Kubernetes documentation. https://kubernetes.io/docs/concepts/storage/volumes/#emptydir (accessed 4 October 2026).
3. Google Cloud. "Container runtime contract: File system access." Cloud Run documentation. https://docs.cloud.google.com/run/docs/container-contract (accessed 4 October 2026).
4. Debian. "The temporary-files directory /tmp is now stored in a tmpfs." Release notes for Debian 13 (trixie), section 5.1.6. https://www.debian.org/releases/trixie/release-notes/issues.en.html (accessed 4 October 2026).
5. The Linux kernel developers. "Tmpfs." The Linux Kernel documentation. https://docs.kernel.org/filesystems/tmpfs.html (accessed 4 October 2026).
6. The Linux kernel developers. "Control Group v2: Memory." The Linux Kernel documentation. https://docs.kernel.org/admin-guide/cgroup-v2.html (accessed 4 October 2026).

Appendix A: Write test

Run with, for example, docker run --rm --memory=200m --memory-swap=200m --tmpfs /scratch:size=1g -v "$PWD/inside.sh:/inside.sh:ro" alpine:3.20 /inside.sh /scratch 250.

The first argument is the directory to write to, the second the number of MiB to write.

#!/bin/sh
cg=/sys/fs/cgroup/memory
[ -d $cg ] || cg=/sys/fs/cgroup
usage() { cat $cg/memory.usage_in_bytes 2>/dev/null || cat $cg/memory.current; }
shmem() { grep -E '^(total_)?shmem ' $cg/memory.stat | head -1 | awk '{print $2}'; }
echo "before usage_mb=$(( $(usage) / 1048576 )) shmem_mb=$(( $(shmem) / 1048576 ))"
dd if=/dev/zero of=$1/big bs=1M count=$2 2>/tmp/dd.err; rc=$?
echo "dd_exit=$rc $(tail -1 /tmp/dd.err)"
echo "file_mb=$(( $(stat -c %s $1/big 2>/dev/null || echo 0) / 1048576 ))"
echo "after usage_mb=$(( $(usage) / 1048576 )) shmem_mb=$(( $(shmem) / 1048576 ))"
df -m $1 | tail -1

Appendix B: Persistence test

Run with docker run --rm --memory=200m --memory-swap=200m --tmpfs /scratch:size=1g -v "$PWD/persist.sh:/persist.sh:ro" alpine:3.20 /persist.sh.

Program 1 writes 150 MiB to /scratch and exits. Program 2 is a separate program that needs a 120 MiB buffer of ordinary memory.

#!/bin/sh
cg=/sys/fs/cgroup/memory; [ -d $cg ] || cg=/sys/fs/cgroup
usage() { echo $(( $(cat $cg/memory.usage_in_bytes 2>/dev/null || cat $cg/memory.current) / 1048576 )); }
sh -c 'dd if=/dev/zero of=/scratch/cache.bin bs=1M count=150 2>/dev/null'; echo "step1_exit=$? usage_mb=$(usage)"
sh -c 'dd if=/dev/zero of=/dev/null bs=120M count=1 2>/dev/null'; echo "step2_exit=$? usage_mb=$(usage)"
rm /scratch/cache.bin; echo "file deleted usage_mb=$(usage)"
sh -c 'dd if=/dev/zero of=/dev/null bs=120M count=1 2>/dev/null'; echo "step2_again_exit=$?"