Thread-Per-Request Overload

Thread-per-request handles each connection or request on its own dedicated OS thread that blocks on I/O until the work finishes. It is simple and intuitive, but when threads are created without bound — or the pool is sized far beyond what the machine can run — concurrency is limited by how many threads the OS can schedule and how much memory their stacks consume.

Why It Happens

The model maps cleanly to how we think: one request, one thread, top-to-bottom code with ordinary blocking calls. Older frameworks and tutorials default to it. Under light load it works perfectly, so teams never revisit it. Then traffic grows, downstream calls get slower, and threads pile up waiting on I/O — but each new request still demands a fresh thread.

Why It Hurts

Each OS thread reserves a stack (often ~1 MB) and competes for the scheduler. A few thousand blocked threads can exhaust memory and spend more time context-switching than working. Because most threads are merely waiting on I/O, the CPU sits idle while the server falls over — a waste of capacity. Without a bound, a traffic spike or a slow dependency triggers unbounded thread creation, then OutOfMemory and a crash, rather than graceful back-pressure. This is closely related to blocking the event loop and to having no back-pressure.

Warning Signs

• Thread count climbs with concurrent requests and never plateaus.
• High context-switch rate and rising memory while CPU stays low.
• OOM or unable to create new native thread errors under load.
• Latency cliffs: fine until a threshold, then sudden collapse.

Better Alternatives

• Bounded thread pool with a queue and rejection policy: cap concurrency and shed or queue excess instead of crashing.
• Asynchronous / non-blocking I/O (event loop, reactive streams, async/await): handle thousands of connections on a handful of threads.
• Virtual threads / green threads / goroutines: keep the simple blocking style while the runtime multiplexes millions of lightweight threads onto few OS threads.
• Back-pressure at the intake so the system signals "slow down" rather than absorbing unbounded load.

How to Refactor Out of It

First, bound it: replace unbounded thread creation with a fixed-size pool plus a bounded queue and an explicit rejection or back-pressure policy, so overload degrades predictably. Then decide the longer-term model. If the language offers lightweight threads (Go goroutines, Java virtual threads, Kotlin coroutines), adopt them to keep blocking-style code while scaling concurrency cheaply. If not, move I/O-bound paths to non-blocking async APIs so few threads serve many in-flight requests. Load-test to confirm the system now plateaus and sheds load instead of collapsing.