Unbounded Cache

An unbounded cache stores entries indefinitely with no maximum size, no eviction policy, and no expiry. It starts as a sensible optimization — keep computed or fetched values around to avoid recomputation — but because nothing is ever removed, the cache grows with every unique key it sees. Over time it behaves like a memory leak whose root cause is "the cache."

Why It Happens

The simplest cache is a plain hash map: check it, and on a miss compute and store. Adding eviction requires choosing a policy, a size, and an expiry — extra decisions that get deferred. In testing the key space is small, so the cache stays tiny and looks perfect. Caches keyed by user input, request parameters, or generated IDs have effectively unbounded key spaces, so in production the entry count keeps climbing.

Why It Hurts

Memory grows until the process is OOM-killed or the GC thrashes trying to manage a huge live set, converting a latency win into reliability incidents. Even before that, a giant cache hurts locality and GC performance. And because entries never expire, the cache serves stale data long after the source changed, causing correctness bugs. The optimization meant to make the system faster instead makes it slower and less reliable.

Warning Signs

• A Map/dict/HashMap used as a cache with no eviction or TTL.
• Cache keyed by unbounded input (IDs, query strings, user content).
• Heap usage that tracks the number of distinct keys seen, not the working set.
• Reports of stale values that only clear on restart.

Better Alternatives

• LRU/LFU cache with a max size (Caffeine, Guava, lru-cache, functools.lru_cache(maxsize=...)): evict the least useful entries.
• TTL-based expiry: entries become invalid after a bounded time, bounding staleness.
• Size- and weight-bounded caches: cap by entry count or memory footprint.
• External cache (Redis/Memcached) with eviction policies and memory limits for shared, large caches.
• Cache invalidation hooks so writes evict affected keys instead of relying on time alone.

How to Refactor Out of It

Replace the raw map with a real cache library configured with a maximum size and an eviction policy (LRU/LFU) plus a TTL. Size the cache to the actual working set, not the full key space, and measure hit rate to confirm the bound is not too tight. Add expiry to bound staleness, and where correctness matters, invalidate entries on writes. For large or shared caches, move to an external store with built-in memory limits. After the change, run a soak test with realistic key cardinality and confirm memory plateaus while hit rate stays acceptable.