Skip to content

fs.mkdir with recursive: true retries without a progress bound: spins forever on /proc, livelocks under concurrent rmdir #66268

Description

@rndjams

Version

v22.23.3 and v24.21.0 (both reproduce)

Platform

Linux 6.8.0-100-generic #100-Ubuntu SMP PREEMPT_DYNAMIC Tue Jan 13 16:39:21 UTC 2026 aarch64 GNU/Linux

Official node:22 and node:24 images (Debian 12 userland). /proc is an ordinary procfs mount with
default options: proc /proc proc rw,nosuid,nodev,noexec,relatime.

Also reproduced independently on x86_64, on GitHub-hosted ubuntu-latest runners with Node 22 and 24,
where the hung job hit the 6-hour ceiling twice.

Subsystem

fs

What steps will reproduce the bug?

$ node -e "require('node:fs').mkdirSync('/proc/zzz', { recursive: true })"

No output, no error, 100% CPU until killed. fs.promises.mkdir and the callback form of fs.mkdir
behave the same way. Dropping recursive: true returns ENOENT immediately.

A second reproduction needs no pseudo-filesystem. On an ordinary disk filesystem (I used ext4; T
below is a directory on it, not a tmpfs or overlayfs), run this loop:

const fs = require('node:fs');
const p = `${T}/parent/child`;
for (let i = 0; i < 5000; i++) {
  fs.mkdirSync(p, { recursive: true });
  fs.rmSync(p, { recursive: true, force: true });
}

while N other processes spin on fs.rmdirSync(T + '/parent/child') and
fs.rmdirSync(T + '/parent'). Latency per call degrades with N:

concurrent removers mean per call worst call calls over 50ms (of 5000)
0 0.006ms 0.1ms 0
8 0.158ms 14.8ms 0
32 2.115ms 168ms 93

(4 vCPUs. Racers must issue the rmdir syscall directly; shell rmdir(1) racers are too slow to win
the race because of fork+exec overhead.)

How often does it reproduce? Is there a required condition?

The /proc case is 100% reproducible on Linux with no special conditions or privileges. It does not
reproduce on macOS or Windows, which have no /proc.

The concurrent-removal case is timing-dependent and scales with contention. The window is narrow on
fast local storage, so it shows up as latency rather than a permanent hang at the numbers above.
Widening the window by running the mkdir side under strace drove it to 67 mkdirat syscalls per
attempted mkdirp, sustained, and 2000 iterations did not finish inside a 60-second timeout.

What is the expected behavior? Why is that the expected behavior?

mkdirSync(p, { recursive: true }) should either create the directory or fail with an errno, in
bounded time. mkdir -p and mkdirSync without recursive both do: the latter returns ENOENT
immediately for /proc/zzz. Recursive mode should not be able to turn a failing mkdir into an
unbounded loop.

What do you see instead?

An unbounded retry loop. strace -f -e trace=mkdir,mkdirat over 5 seconds of the /proc case shows
22,872 mkdirat calls in a repeating 2-call cycle:

mkdirat(AT_FDCWD, "/proc/zzz", 0777) = -1 ENOENT (No such file or directory)
mkdirat(AT_FDCWD, "/proc", 0777)     = -1 EEXIST (File exists)
mkdirat(AT_FDCWD, "/proc/zzz", 0777) = -1 ENOENT (No such file or directory)
mkdirat(AT_FDCWD, "/proc", 0777)     = -1 EEXIST (File exists)
...

Additional information

Mechanism

MKDirpSync in src/node_file.cc
treats ENOENT as "the parent does not exist yet", pushes the failed path back onto its stack, pushes
the parent ahead of it, and retries:

case UV_ENOENT: {
  std::string dirname =
      next_path.substr(0, next_path.find_last_of(kPathSeparator));
  if (dirname != next_path) {
    req_wrap->continuation_data()->PushPath(std::move(next_path));
    req_wrap->continuation_data()->PushPath(std::move(dirname));
  } else if (req_wrap->continuation_data()->paths().empty()) {
    err = UV_EEXIST;
    continue;
  }
  break;
}

ENOENT is the only errno that re-queues work. EACCES, ENOSPC, ENOTDIR and EPERM return
immediately, and everything else reaches default:, where a failing stat ends the walk. So the loop
terminates only where mkdir returning ENOENT implies a missing parent, and POSIX does not
guarantee that.

Two cases where it does not hold:

  • Under /proc, mkdir returns ENOENT for a name that can never be created, even though the parent
    exists. Each cycle pops /proc/zzz (ENOENT), pushes it back with /proc, pops /proc
    (EEXIST, stat confirms a directory, falls through), and finds /proc/zzz still on the stack.
  • Under concurrent removal, the ENOENT is real but transient: a peer removed the parent after the
    last check, so the retry recreates it, and the child create can miss again.

Neither MKDirpSync nor MKDirpAsync records whether an iteration made progress, so there is no
state that can end the loop.

For contrast, /sys does not hang: mkdir there fails with EROFS, which reaches default: and
terminates. /etc/hostname/zzz returns ENOTDIR. Only the ENOENT path re-queues.

Impact

A recursive mkdir on a path the caller does not fully control, or one contended by another process,
can go from microseconds to unbounded. On the sync path the spin is inside a C++ call, so no JS
timeout, AbortSignal or signal handler interrupts it; only SIGKILL from outside the process ends
it. Running the one-liner under docker run and pressing Ctrl-C three times is enough to see this:
the CLI gives up forwarding signals and detaches, and the container keeps running and keeps spinning.
The async path occupies a libuv threadpool worker the same way.

I hit the /proc case in CI, where a test used a path under /proc as an unwritable fixture. Nothing
crashed and nothing was logged, so the run was killed only by GitHub Actions' own 6-hour job limit. I
did not notice the hang itself: what surfaced was every later job being refused for billing, because
those 6 hours had exhausted the account's Actions budget. The test had always passed on macOS, where
/proc does not exist.

Possible fix

Bound the retry on lack of progress. If a path is popped a second time with no successful mkdir in
between, return the original error rather than pushing its parent again. The uncontended path stays as
cheap as it is today, and both MKDirpSync and MKDirpAsync then terminate regardless of what a
filesystem returns.

A retry cap would also stop the race case, but it leaves /proc looping unless the cap is small
enough to apply there too, which makes it the same fix with a number attached.

I am happy to open a PR with tests for both cases if that direction seems right.

Related

Two earlier reports of this loop, both on Windows, both fixed by correcting one filesystem's errno
rather than by bounding the loop:

AI-assisted. Reproduced and measured in a Linux container before filing; the one-liner cannot
reproduce on macOS, which has no /proc.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions