On BEAM/ERTS limits

The atom table exhaustion got another round of attention in the BEAM community recently. It is far from the only way to render a node unusable, and not even the most interesting one.

A couple of the others I ran myself into.

When a process dies and its state is large, the crash report carries that state into the log, and the act of formatting and writing it can be heavier than whatever killed the process in the first place. Simply logging the stacktrace can take down a node by making it try to tell you why a process went down. The cure is the usual one, keep state out of the parts that end up in error_logger. It can be done by wrapping the state in a fun, which by the way will prevent exposing secrets and sensitive info the process might hold.

The one I want to write here is old, and the first reports can be traced to 2009. I hit it while setting up a fresh Arch Linux VM for Knative Eventing RabbitMQ integration, where it cost me enough time before I understood what I was looking at. The BEAM can run out of memory before it finishes starting, and the amount it grabs depends on a number one would think about last.

2009 — mailing list thread

In 2009 Vincent de Phily posted to erlang-questions that a bare shell was eating close to 300 MB (thread):

1> erlang:memory().
[{total,302822080},
 ...
 {system,302333044},

He traced it to the open-file limit. Sweep ulimit -n from a million down to fifty and the system memory tracks it, with jumps at the powers of two:

999999 301852400
500000 152120480
200000  74654496
 10000   6775136
    50   2629488

Per Hedeland gave the explanation in the next message. The runtime allocates a port table sized to the next power of two at or above sysconf(_SC_OPEN_MAX), and it does this at startup whether or not you ever open a port. ERL_MAX_PORTS lets you set it directly, with a floor of 1024 and a ceiling he read off as 2^28. He pointed at init_io() in erts/emulator/beam/io.c.

That was R13. The interesting question is whether it is still true.

2022 — My Knative story

The setup was a single-node operator-managed RabbitMQ on a fresh Arch box, part of getting Knative Eventing RabbitMQ running. It kept landing in CrashLoopBackOff (cluster-operator#959). Stripping the entrypoint down to a shell showed that the first rabbitmq-server invocation exited cleanly and every one after it printed Killed with code 137. OOM.

The cause was the file-descriptor limit again, but not the host’s. The host and the container disagreed about nofile by three orders of magnitude:

Arch host:                     ulimit -Hn  524288
Rabbit on Arch inside k8s:     ulimit -Hn  1073741816
Ubuntu host:                   ulimit -Hn  1000000
Rabbit on Ubuntu inside k8s:   ulimit -Hn  1048576

The Arch host I was running already carried an enormous default limit, and the kind/containerd path handed the pod 1073741816. The Ubuntu host ran the same image and got a sane 1048576, which is why it never crashed. The OOM was distro-shaped: a Ubuntu-based image inheriting an Arch host's idea of a reasonable fd limit through the container runtime.

The memcg report when beam.smp got killed:

memory: usage 2097152kB, limit 2097152kB, failcnt 140
...
anon 2129944576
...
Killed process 1917152 (beam.smp) total-vm:4694628kB, anon-rss:1647628kB

Exactly 2 GiB of cgroup limit, almost all of it anonymous, hit while the node was still coming up.

The fix was either lowering ulimit -n in the container or setting ERL_MAX_PORTS to something like 50000, which is what we ended up recommending in the docker-library thread.

2026 — where we are today

At the time I left a note on the issue saying the mechanism looked like the 2009 one and that I would check whether it still held. Let’s finally look at the code, against master (OTP 30.0-rc0, commit 0c452ae, June 2026).

Here is [erts_init_io](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/io.c#L2971) in io.c:

if (!port_tab_size_ignore_files) {
    int max_files = sys_max_files();
    if (port_tab_size < max_files)
        port_tab_size = max_files;
}

...

if (port_tab_size > ERTS_MAX_PORTS)
    port_tab_size = ERTS_MAX_PORTS;
else if (port_tab_size < ERTS_MIN_PORTS)
    port_tab_size = ERTS_MIN_PORTS;

erts_ptab_init_table(&erts_port, ..., port_tab_size, ...);

Unless you set ERL_MAX_PORTS or pass +Q (either flips port_tab_size_ignore_files), the fd limit raises the table's floor, up to the [ERTS_MAX_PORTS](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/erl_node_container_utils.h#L248) cap of 2^27-1, with [ERTS_MIN_PORTS](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/erl_port.h#L52) (1024) at the low end. sys_max_files() still resolves to [sysconf(_SC_OPEN_MAX)](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/sys/common/erl_poll.c#L2089), through erts_check_io_max_files() and erts_poll_max_fds() in erl_poll.c, the same syscall Per Hedeland named.

The allocation itself is in [erts_ptab_init_table](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/erl_ptab.c#L359) (erl_ptab.c). It rounds the requested size up to a power of two and then calls erts_alloc_permanent_cache_aligned:

bits = erts_fit_in_bits_int32(size-1);
size = 1 << bits;
...
tab_sz = ERTS_ALC_CACHE_LINE_ALIGN_SIZE(size*sizeof(erts_atomic_t));
alloc_sz = tab_sz;
if (!legacy)
    alloc_sz += ERTS_ALC_CACHE_LINE_ALIGN_SIZE(size*sizeof(erts_atomic_t));
ptab->r.o.tab = erts_alloc_permanent_cache_aligned(atype, alloc_sz);

2009 vs 2026

In 2009 the table was an array of Port structs. init_io allocated [erts_max_ports * sizeof(Port)](https://github.com/erlang/otp/blob/OTP_R13B04/erts/emulator/beam/io.c#L1218), sized to the fd limit and rounded to a power of two, over a hundred bytes per potential port up front. The 2009 email thread was about R13B01, but GitHub only goes back to R13B04, so that is what I link here. Today the Port structs are allocated on demand by [erts_ptab_new_element](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/erl_ptab.c#L500), and only the index array and a free-id ring are preallocated, so it consumes less memory now. The shape did not change: still sized to the fd limit, capped at ERTS_MAX_PORTS, still allocated at startup.

For the pod’s 1073741816: it exceeds ERTS_MAX_PORTS, so io.c clamps it to 2^27-1 and erts_ptab_init_table rounds up to 2^27 slots. Two arrays, one word each, on 64-bit:

2^27 slots * 8 bytes * 2 = 2 GiB

The init loop writes every slot, which drives the resident memory up to the full table size. That resident 2 GiB is the memory: usage 2097152kB, limit 2097152kB and anon 2129944576 from the OOM report above.

ERL_MAX_PORTS (or +Q) flips[port_tab_size_ignore_files](https://github.com/erlang/otp/blob/0c452aefa17a6740b7ada87013f19124ab54bb10/erts/emulator/beam/erl_init.c#L1335) and decouples the table from the fd limit. The first attempt to fix this was in the RabbitMQ docker image (docker-library#545), and it later moved into rabbitmq-server itself (#5684).

Lessons

• ERTS still preallocates the port table at startup, sized to the fd limit, capped at ERTS_MAX_PORTS (2^27-1) and rounded up to a power of two, unless ERL_MAX_PORTS or +Q is set. True in R13, still true on OTP 30 master.
• In a container this has a side effect worth knowing. The runtime sizes the table against the fd limit it sees at startup, which kind/containerd can set far higher than the ulimit -n you would see in a normal shell, and the pod usually runs under a memory cgroup. A limit like 1073741816 clamps to 2^27 ports and brings about 2 GiB resident at boot, enough to OOM a 2Gi pod before it serves anything. Running RabbitMQ under a 2Gi limit is not realistic anyway, but the finding generalizes past RabbitMQ: any ERTS-based service preallocates this table from the fd limit, and a tight enough memory cgroup turns it into an OOM. Set ERL_MAX_PORTS explicitly to keep the table off the fd limit.
• Atom table exhaustion is the usual example of running a node out of resources. Crash-dumping a large process state, and preallocating a large port table under a tight memory cgroup, are two others.