Four MCP servers and the leak that wasn’t

A production Rails API spent four weeks over its Heroku memory quota. It was not a memory leak.

SWARECO found the cause by putting four observability tools into a single session and reading them against each other.

The endpoint responsible was consuming 21% of total application time while allocating almost no Ruby objects. This is the signature of bytes moving off-heap, where no Ruby profiler can see them. Four weeks of allocation work across five pushes had not moved the number. Two days after we correlated the tools, the problem was fixed.

This is the whole story, including the part where we were confidently wrong for three of those weeks.

The client is unnamed. This is an account of a defect in someone’s production system, not a case study. Everything technical is exact.

The symptom, and why the obvious reads were wrong

The application threw Heroku R14, Memory quota exceeded, continuously on the web tier. The API was running Rails 7.2.3 and Ruby 3.4.1. Here is a representative 24-hour period.

	web (Performance-M, 2,560 MB)	worker (Standard-2X, 1,024 MB)
Shape	climbs monotonically, resets only on the nightly deploy	flat, oscillating
Peak	189.6% of quota — 4,854 MB	121.7% / 1,246 MB
Average	60.7% / 1,554 MB	71.7% / 735 MB
Memory errors	2,236	89

The web dynos’ memory usage grew without bound. It only ever came back down when a deploy restarted the dynos. The workers were healthy, noisy but with no upward trend.

This ran for four weeks instead of paging someone at 3 a.m. because nothing crashed. Sentry showed no SIGKILL and no OOM kill across the entire period.

An R14 error on Heroku is not a crash. The dyno exceeds its quota, starts swapping to disk, and gets slower. The application degrades. Requests still return. Users experience a slow product, not an outage, and no alarm has a threshold for “slow.”

The obvious read for a Rails app with climbing memory is allocation churn. Think N+1 queries, per-request object bloat, or serializers rebuilding the same structures. That read was correct about the application in general. It was wrong about this specific problem. Both things can be true at once, which is exactly what makes this class of bug expensive to solve.

Four weeks of being right about the wrong thing

We spent three weeks and three pushes cutting allocations. Every one of those changes was real, measured, and shipped. None of them touched the R14 errors.

The wins were not marginal.

Endpoint	Objects before → after	Bytes before → after
list endpoint A	313,518 → 92,293	34.27 MB → 8.98 MB (−74%)
list endpoint B	186,383 → 96,597	19.43 MB → 10.55 MB (−48%)
detail endpoint	15,073 → 13,434	1.42 MB → 1.27 MB (−11%)

Inside list endpoint A, a phone-number formatting gem went from 10.65 MB to 438 kB per request. It had been re-running per row instead of once per tenant. We removed N+1s, memoized shared associations, pushed DISTINCT into SQL, and switched JSON encoding to Oj. We stopped materializing two Postgres-generated tsvector columns per row and eliminated an INSERT-on-GET.

We built a repeatable profiling harness with fixed-seed data using memory_profiler and derailed_benchmarks. This covered seven endpoint scenarios behind a rake task. We also added rspec-memory allocation-budget specs so the gains could not silently regress.

It was good, correct work. R14 continued exactly as before.

We also ruled out the allocator, which is the other standard answer. jemalloc was not just installed but confirmed loaded on both tiers via LD_PRELOAD. MALLOC_ARENA_MAX was correctly left unset. The fragmentation lever had already been pulled before we arrived. It wasn’t that either.

The wrong turn, in full

Then we misdiagnosed it. The mistake is instructive because of which tool produced it.

A 90-second window from heroku logs --dyno worker showed worker RSS climbing from 450 MB to 844 MB. Read on its own, that looks like a creeping floor, memory arriving and not leaving. We concluded the workers were the problem and started looking there.

The 24-hour metrics graph disproved it. The workers were flat over any window long enough to have a shape. That 90-second sample had caught a set of analytics Sidekiq jobs, each adding 150 to 280 MB by materializing millions of ActiveRecord rows. These were transient spikes that jemalloc released cleanly. The jobs were worth fixing on their own merits, but they were not the cause. They were not even on the tier that had the problem.

The lesson went into our investigation document in the same words we would use now: don’t diagnose a slope from a 90-second window. A short sample of a sawtooth pattern is indistinguishable from the leading edge of a ramp.

Why no single tool was going to find this

Each tool answers exactly one question well, and answers every other question misleadingly.

• memory_profiler shows which Ruby objects a request allocates. It cannot see memory Ruby never allocated.
• Heroku metrics report how much resident memory the dyno holds. It cannot say what is in it.
• Sentry tells you what is raising errors, and how often. A byte-streaming problem raises almost nothing.
• Scout shows where wall-clock time goes, per endpoint.
• Better Stack logs which requests moved how many bytes.

The bug lived in the seam between them. Every individual reading was accurate, and every individual conclusion drawn from it was wrong. The decisive fact was a relationship between two tools’ outputs, and no single tool holds two tools’ outputs.

Wiring four MCP servers into one session

Four of the five are MCP servers: Scout, Sentry, AWS, and Better Stack. The fifth was the heroku CLI.

That split turns out to matter, and not just as bookkeeping. The four that answered in-session got asked constantly, in whatever order a question happened to need. Heroku got asked when someone remembered to go and look. This is precisely how the 90-second sample became load-bearing for two days longer than it should have.

A tool you have to go and ask by hand is a tool you check late, and check less.

What correlation actually looked like was one question, four answers, in one place, cheap enough to repeat. This endpoint is slow, is it allocating? Is it erroring? How many bytes is it moving? Is the dyno holding them? Answering that used to mean four browser tabs, four query languages, and enough context-switching that you tended to ask the cheap version of the question instead. When all four answer in the same session, the correlation stops being a project and becomes a reflex.

The first thing that reflex produced was a negative result from the Ruby heap graph. Allocation and free were balanced, and the live-object band was flat. Ruby was not holding more objects over time. It never had been.

The signal that broke it open

Flat Ruby heap, climbing RSS. Those two facts are only compatible with one conclusion: the growth is off-heap. It’s native memory, outside anything a Ruby object profiler can instrument. Every measurement we had spent three weeks improving was, by construction, blind to it.

Scout localized it immediately. Here are the endpoints ranked by their share of total application time.

Endpoint	% total time	p95	max Ruby allocations
active_storage/blobs/proxy/show	21.1%	1,057 ms	31,500
list endpoint B	16.0%	629 ms	1,000,000
list endpoint C	16.2%	816 ms	1,600,000
filter endpoint	4.5%	393 ms	3,100,000

Read the first row against the others. The ActiveStorage blob proxy consumed more application time than any other endpoint while allocating roughly 30 times fewer Ruby objects than its nearest neighbor. Work that costs a fifth of your application’s time but allocates nothing is not doing Ruby work. It is moving bytes.

Sentry supplied the mechanism. On that same controller, we saw ClientDisconnected ×394 and Slow DB Query ×621 on the proxy path. This matched a heavy H27 (client interrupted) band in the Heroku logs. Clients were walking away mid-download, leaving streams abandoned in flight. Sentry also gave us the negative that explained the four weeks: no OOM, no kill. The app was degrading, not crashing.

Better Stack sized the problem, which in turn determined the shape of the fix. During one live R14 event, the web dyno’s RSS was at 98.5% of quota while workers were flat at 74.5%. The ActiveStorage image proxy accounted for ≈491.5 MB, about 87% of all bytes through the web tier.

Widened to a 3.5-day window, the proxy served 23,497 requests, totaling about 27 GB. The average size was 1.23 MB each, and they were essentially 100% full-size original images. Over the same period, resized variants through the proxy accounted for just 16 requests and half a megabyte.

Then we asked the question that changed the fix: who is asking for 27 GB of full-size originals? It wasn’t users. It was automated clients. A third-party platform’s image crawler accounted for roughly 28%, Googlebot for about 1.5 GB, and a spread of datacenter scrapers. All of them were holding permanent URLs the application had already handed out and could not take back.

The cause

The app set resolve_model_to_route = :rails_storage_proxy. The S3 bucket is private, so objects cannot be served directly without expiring signed URLs. The proxy route gave stable, auth-free image URLs plus on-the-fly variant resizing. It is a reasonable choice, and it was made months before any of this began.

The consequence is that every image byte in the application travelled from S3, to the web dyno, to the client. Full-size originals streamed through the same Puma processes that were serving JSON, at 10 threads per dyno. This created off-heap buffers with no Ruby allocations, making them invisible to the profiler but entirely real to the kernel.

The fix, and the guardrail

The naive fix is to use :rails_storage_redirect, which hands out expiring signed S3 URLs. SWARECO rejected it, and the reason is the trap in this particular fix. The web frontend appends ?variant= to image URLs, which breaks a signature and returns a 403 error. The mobile client caches by URL, so rotating signatures would turn every cache hit into a miss. The naive fix would have traded a memory problem for a broken image pipeline on two clients.

A CDN distribution was already live and was the preferred path in the app’s own URL helper. The cutover was simply unfinished. The fix had two halves.

1. Finish the cutover. We migrated every remaining server-side emitter still calling .url or rails_blob_url onto the CDN helper. This included JSON image fields, three mailer templates, and the PDF and print templates. All changes were tested red-first. The awkward one was a PDF renderer that builds its own context and therefore doesn’t load app/helpers. The helper raised a NoMethodError until we gave it a scoped controller.

2. Redirect the traffic we could not reach. Finishing the cutover stops the app from emitting new proxy URLs. It does nothing about URLs already baked into third-party systems, cached feeds, and crawled pages, the source of the 27 GB of traffic. So we prepended a module onto ActiveStorage’s own ProxyController#show. If an image blob was requested and the CDN was configured, we issued a 302 redirect to the CDN. The dyno performs a near-zero-cost redirect instead of pumping a megabyte. Non-image blobs, or environments with no CDN configured like dev and test, fall through to the gem’s default streaming via super. This drains the long-tail traffic immediately on deploy, regardless of who holds the URL or how old it is.

Then we added the guardrail, because the failure mode of this fix is silent regression. We implemented a policy object that fails the production boot if the CDN URL is unset. One check protects every delivery path, so image delivery can never quietly fall back to streaming through the web tier again. It is skipped during asset precompilation and is a no-op outside of production.

For scale, a thumbnail variant from the CDN is 13 kB against a 337 kB original PNG, WebP-converted, on a one-year public cache. Blob keys are content-addressed and immutable, so cache invalidation is never needed. A changed image is a new blob with a new key and a new URL.

Both fixes were merged two days after the four tools were first read against each other. The four weeks before that were spent on a hypothesis that three of the five tools actively supported.

What this generalises to

This was a classic case of the symptom and cause being in different measurement domains. The symptom lived in resident memory. The cause lived in bytes on the wire. Nothing in the Ruby object graph connected them, so every tool aimed at the Ruby object graph returned a clean bill of health while the number kept climbing.

Three things SWARECO would now check first, in this order:

1. A flat Ruby heap with climbing RSS means stop profiling allocations. The answer is off-heap: native buffers, image processing, byte streaming, or a C extension. Continuing to optimize allocations at that point is measurable, publishable, and irrelevant. We have the −74% metric to prove it.
2. A high share of application time with near-zero allocations is a byte-movement signature. It is one of the few readings that is nearly unambiguous. It takes a tool that reports time and allocations side by side to see it at all.
3. Ask which clients hold your URLs. A fix that changes what your app emits does not reach URLs already in someone else’s database. We found that out from traffic analysis, not from code, and it turned a one-PR fix into a two-PR fix.

On the tooling, the honest claim is not that an AI found this bug. The correlation was the work, and the correlation was human. What four MCP servers in one session changed is the cost of correlating. It went from four tabs, four query dialects and a context switch, down to something cheap enough to do repeatedly on a hunch. The fifth tool, the one behind a CLI, is the control group. It is where our worst mistake came from, because it was the one we sampled instead of watched.

What we changed so it gets caught next time: the profiling harness and allocation-budget specs stayed, as they guard real regressions, just not this one. The boot-time delivery guardrail is new. And the investigation document keeps its own retraction, the 90-second window, what we concluded from it, and why that was wrong, sitting above the corrected findings where the next person will read it first.