Observability Feedback Loop: How One Quietly Burned $750/Day in Our Own Infra

TL;DR: An observability feedback loop is when the act of monitoring a system produces data that becomes more for the system to monitor, so volume compounds instead of settling. One ran inside our own infrastructure and cost about $750 a day, climbing, with no traffic spike and no deploy. Our own agent triangulated it on day one, reading the cost spike and the error flood together and naming the service that drove both. We waved it off because the product was young, then spent 48 hours and three engineers re-deriving the same answer. The fix was to cut the loop at its cheapest edge and starve the amplifiers, and the cost collapsed to near zero the same day it shipped.

Key takeaways

• An observability feedback loop forms when monitoring output (error logs) is fed back in as monitoring input (ingestion), so each cycle creates the next.
• The tell is shape: compounding, super-linear cost growth with no upstream change. A real workload increase steps or ramps, then plateaus.
• You cannot find a single root cause in a loop, because every effect is also a cause.
• Break it at the cheapest edge first (stop generating the logs that feed it), then starve the amplifiers: verbose diagnostics, unfiltered logs, long job timeouts, and missing log retention.
• Detecting the symptom is table stakes. The value is connecting the symptom to its cause across signals, and trusting that connection enough to act.

What is an observability feedback loop?

An observability feedback loop is a self-amplifying cycle in which monitoring a system generates new data that the same system then has to monitor. A failing job writes an error log, the logging pipeline ingests that error log as new data, ingestion is work that re-runs the job, the job fails again, and it writes another error log. The pipeline feeds itself, and data volume and cost compound instead of settling.

This is the story of the week that pattern caught us, told us plainly through our own product, and we waved it off anyway.

We built the tool. Then we didn’t trust it. That’s the whole story.

In Sherpa’s early days we did the thing every honest builder does: we pointed our own product at our own infrastructure. We ingested our own logs, ran our own detectors over them, and quietly didn’t trust the output yet. It was early. Surely it wasn’t ready to catch anything real.

It was. It caught something real, told us plainly, and we dismissed it. Then we spent 48 hours and three engineers re-discovering exactly what it had already said.

The first symptom: a cost going up for no reason

The first symptom was boring, which is exactly why it is dangerous. Our cloud cost for log storage started climbing on its own. No traffic spike, no new customer, no deploy. Just a line bending the wrong way, about $750 a day and rising. Then the pipeline that turns raw telemetry into insight started lagging. Then a few jobs in it flipped red.

None of these pointed at each other. A storage graph, a slow pipeline, and a failing job look like three unrelated tickets. We opened them as three unrelated tickets. That was the first mistake. The second one is the one I think about more.

We asked Sherpa. Sherpa answered. We ignored it.

Somewhere early in the hunt, almost as a test, we asked our own agent what it made of the mess. It triangulated. It read the cost curve and the error flood together, connected the two, and named the service that was driving both, our own ingestion eating its own error logs. Then it pointed at the kind of fix that would actually break the cycle.

And we waved it off.

Not because it was vague, but because it was ours, and it was early, and the bug felt too deep for a young product to have genuinely understood. “No way it caught something this subtle.” We filed the agent’s answer under ‘nice try’ and went back to doing it the hard way.

Why a feedback loop is so hard to root-cause

So we did it the hard way. Three engineers, two days, all focused on the same problem because the meter was running at $750 a day and climbing. And it was genuinely hard, because the cause was also a symptom. Every effect was also a cause.

The job failed, so it wrote an error log. The error log landed in CloudWatch. Our pipeline ingests CloudWatch logs, that is its job, so it picked the error up as new data. New data is work, and the work re-ran the job. The job, still broken, failed again, and wrote another error log. Round and round, each lap a little bigger, with two amplifiers pouring on fuel: a diagnostic setting on our collector turned all the way up (a line per batch), and a noisy class of internal request logs we meant to filter but had not. Add jobs allowed to hang up to 48 hours before timing out, so a single stuck one could spew for two days.

We kept asking “what is causing the errors?” when the honest answer was “the errors are causing the errors.” You cannot find a root cause when the graph is a circle.

The loop, drawn straight

A job errors, the error becomes a log, we ingest the log, ingesting is work, the work re-runs the job, the job errors again.

The tell we missed was the shape of the growth. A real workload increase steps or ramps and then plateaus. A feedback loop compounds, because each cycle seeds the next.

How we broke the log feedback loop

Once we finally saw the curve for what it was, the fix was almost embarrassingly simple. We cut the loop at the cheapest edge and starved it:

• Stop treating an empty run as a failure, so it no longer emits an error log.
• Turn the diagnostic firehose down and off the hot paths.
• Filter the noisy internal logs at the door, before ingestion.
• Cap job timeouts from days to minutes.
• Put real retention on the log groups so nothing accumulates unbounded.

The cost collapsed the same day the fix shipped, from about $750 a day back to roughly nothing, lower than before, because we had also cleared out the noise.

Then we went back and read what Sherpa had said

Here is the part that changed how we build. Once we had fixed it ourselves, someone scrolled back up to the answer we had dismissed two days earlier. It was right. It had named the error flood, connected it to our own ingestion, and pointed at the fix, on day one, while we were still opening it as three separate tickets.

We had the answer. We just did not trust the thing that gave it to us. Two days and three engineers was the price of that distrust.

That is not a humblebrag, it is the opposite. We got caught by the boring version of the exact problem we sell against, and we ignored our own product solving it in real time. The lesson was not “our agent is magic.” It was simpler and more uncomfortable: we had built something we did not yet believe in, and it was further along than we were. Now, when Sherpa flags something that sounds too deep to be true, we check it before we dismiss it. We learned that the expensive way.

We ran this post-mortem on our own environment, in the open. The playbook above works whatever tools you use. If you’d rather have something watching your telemetry for loops like this so you don’t have to re-derive them by hand, that is the kind of thing we are building Sherpa to do.

FAQ

What is an observability feedback loop?

It is when the act of monitoring a system generates data that becomes more for the system to monitor. Error logs get ingested, ingestion does work, the work produces more error logs. The pipeline feeds itself and volume compounds instead of settling.

Why are observability feedback loops so hard to diagnose?

Because every effect is also a cause. There is no single root to find. The failing job, the rising cost, and the slow pipeline are all the same loop seen from different angles. Standard root-cause questions assume a starting point a loop does not have.

How do you tell a feedback loop from a real traffic spike?

Shape. A genuine workload increase steps or ramps and then plateaus. A feedback loop compounds, because each cycle is larger than the last and seeds the next one. Super-linear growth with no upstream change is the tell.

How do you break a log feedback loop?

Cut it at the cheapest edge first, usually by stopping the logs that feed it, for example not treating an empty run as an error. Then starve the amplifiers: turn down verbose diagnostics, filter noisy logs at ingestion, and cap job runtime and log retention so nothing accumulates unbounded.

Can an AI observability agent catch a feedback loop?

Yes. Our agent triangulated it early, reading the cost spike and the error flood together and naming the service that drove both, our own ingestion eating its own error logs. The detectors lit on the symptom; the value was the cross-signal connection. We waved it off because the product was young, then spent two days re-deriving the same answer by hand.