MongoDB File Descriptor Exhaustion: ulimit -n Thresholds, Collection/Index Proliferation & systemd Hardening
Diagnosing OS file descriptor exhaustion under large collection counts and connection scaling; configuring systemd LimitNOFILE and limits.conf.
Operational Verification & Reliability Metadata
The error string is a symptom, not the root cause. Do not restart the server blindly. Run the diagnostic queries below to pinpoint the actual failure mode.
▸Symptom
mongod drops inbound socket handshakes and logs:
Failed to accept socket: Too many open files
WiredTiger error (24): handle-open: open: Too many open files▸What You See in Production
Database rejects incoming connections. File operations halt across collections and index files.
▸Root Cause Matrix
| Probability | Root Cause | Description |
|---|---|---|
| %60 | Default `ulimit -n 1024` Exhaustion | WiredTiger maintains dedicated file descriptors per collection and index. |
| %30 | Socket File Descriptor Starvation | Spikes in concurrent sockets consume available process descriptors. |
| %10 | Missing systemd LimitNOFILE | systemd limits override limits.conf without explicit overrides. |
▸EMERGENCY RUNBOOK (0-5 min)
# 1. Inspect live process limits
cat /proc/$(pidof mongod)/limits | grep "Max open files"
# 2. Elevate descriptors on running process via prlimit
sudo prlimit --pid=$(pidof mongod) --nofile=64000:64000▸PERMANENT BLUEPRINT
- 1systemd Hardening: Set
LimitNOFILE=64000in systemd unit overrides. - 2limits.conf Tuning: Add
mongod soft/hard nofile 64000.
▸Verification
cat /proc/$(pidof mongod)/limits | grep "open files"Other Database Error Triage Playbooks
Other battle-tested diagnostic guides for high-severity database errors.
MongoDB "not master" / NotWritablePrimary Fix
Troubleshooting MongoDB "not master" (legacy < 5.0) and NotWritablePrimary (code 10107 >= 5.0) during replica set failovers, connection string topology URIs, retryWrites, and oplog sizing.
MongoDB E11000: Duplicate Key Error Triage & Mitigation
Diagnosing E11000 errors caused by concurrent insert races, index verification, and implementing atomic upsert with $setOnInsert.
MongoDB MongoNetworkError & HostUnreachable Triage
Addressing driver network timeouts, topology change failovers, invalid SRV connection strings, and serverSelectionTimeoutMS tuning.