Introduction
Software products almost always have people intentionally trying to break them. Whether it's a dedicated QA team, beta users, or internal testing, somebody is constantly validating the experience before it reaches production. Bugs get reported, workflows get questioned, and small frustrations become feature requests. Over time, the product improves because people are continually using it and providing feedback.
Observability platforms don't get that luxury.
The people consuming the platform every day become the closest thing it has to a QA team, whether they realize it or not. Every confusing dashboard, noisy alert, missing context, difficult investigation, or unnecessary workflow becomes feedback about the platform itself. The challenge is that very little of that feedback ever finds its way back into improving the platform.
3000 Pages
One of the experiences that shaped how I think about observability happened while I was working in an Operations Center.
A major outage starts. I'm the only engineer on shift.
Within a few seconds the alerts begin firing and my phone starts going off almost immediately.
At the same time, I have to understand what's actually broken. Is this a single service or something much larger? Which engineering team owns it? Does this warrant declaring an incident? Which engineers should be paged? Tickets need to be created. The incident manager needs to be notified. Internal communications need to go out. Executive email distributions may need to be updated. Public status pages may need to be published. All of those things are happening while I'm still trying to understand what I'm looking at.
Meanwhile my phone just keeps going.
Ding....Ding....Ding....Ding.
I can already see nearly 3,000 pages queued behind the ones I'm receiving. I'm already actively investigating the outage. Every new notification is competing for the same attention I need to understand what's actually happening.
Eventually I turn my phone off.
Not because the alerts are wrong, but because they're no longer helping. At that point they're simply pulling my attention away from the work that actually needs to get done.
After incidents like that I'd usually go back to the engineers responsible for the observability platform. We'd talk through what happened, which alerts helped, which ones didn't, where context was missing, and where dashboards slowed the investigation instead of speeding it up. Sometimes improvements were made and sometimes nothing changed.
After you've had that experience enough times, it's surprisingly easy to become frustrated with the platform. Eventually that frustration starts becoming resentment. To be honest, I think some of that resentment is probably warranted depending on how feedback is handled.
After you've had that experience enough times, it's surprisingly easy to become frustrated with the platform. Eventually that frustration starts becoming resentment.
The platform owners weren't ignoring the feedback. They were responsible for virtualization, storage, automation, Kubernetes, maintenance windows, production issues, security updates, and a dozen other systems competing for their attention. Improving the observability platform wasn't the only thing on their plate. Many weeks it probably wasn't even on the radar.
That's something I understand much better now that I've been on both sides of that conversation.
Your Users Are Your QA Team
One of the easiest traps for an observability platform owner to fall into is becoming disconnected from the people actually using the platform every day. When you're responsible for keeping the platform healthy, it's natural to experience it differently than everyone else. You're thinking about upgrades, storage, authentication, exporters, alerting pipelines, and everything else required to keep it operational. The engineers using the platform experience something entirely different. They experience whether the dashboard answered the question they were trying to ask, whether the alert gave them enough context to act, and whether the platform made an already stressful outage easier or harder to navigate.
That's exactly why their feedback is so valuable.
They're stress testing the platform under real production conditions every single day. They discover confusing dashboards, noisy alerts, broken workflows, documentation gaps, and all the little frustrations that are nearly impossible to find from the platform owner's perspective. The people relying on the platform every day become the feedback loop.
I don't think the cadence really matters. Weekly meetings, monthly syncs, office hours, or another predictable process can all work. What matters is creating a reliable way for those engineers to influence how the platform evolves. Those conversations rarely need much structure because the people using the platform almost always arrive with something they want to improve. Those aren't complaints. They're product feedback.
Those conversations rarely need much structure because the people using the platform almost always arrive with something they want to improve. Those aren't complaints. They're product feedback.
Sometimes the people providing that feedback become some of the platform's main contributors. That's actually how I got my hands dirty with observability. It didn't start with ownership of the platform. It started by using it every day, becoming frustrated by parts of the experience, suggesting improvements, and eventually helping implement them. Looking back, I think that's one of the healthiest feedback loops an observability platform can have because the people improving it are often the same people depending on it during production incidents.
Closing Thoughts
One of the biggest shifts in my thinking happened when I stopped viewing the people using an observability platform as consumers and started viewing them as partners in improving it.
Your observability platform may not have a dedicated QA team. It has engineers using it under real production conditions every single day. They're the first people to discover when an alert isn't actionable, when a dashboard doesn't answer the question it should, or when a workflow creates more friction than it removes.
The platform is already being tested every day. The organizations that improve the fastest are usually the ones that learn to treat that feedback as one of the platform's most valuable inputs.