Reliability Starts Before Production: A Site Reliability Engineer's Perspective on Building Resilient Systems

When systems fail, it is easy to focus on what happens next. How quickly can the issue be found? How fast can services be restored? How quickly can teams get everything back to normal?

For Ayoub El-Moussaoui, Site Reliability Engineer at Mercedes-Benz.io, those questions matter. But they are not where reliability begins, they start long before production. The real work happens earlier: in the decisions teams make when designing systems, defining processes, and creating the visibility needed to understand what is happening when things do go wrong.

Ayoub El-Moussaoui

Site Reliability Engineer|MB.ioneer since March 2023

Fun fact:

You’ll usually find him on a mountain bike whenever he’s away from the keyboard.

In this edition of Tecknowledgy, Ayoub shares why observability has become one of the most important foundations of reliability, how AI is changing the way engineering teams work, and why simplicity often remains the most effective strategy for building resilient systems.

Reliability starts long before production

Throughout his experience working across SRE, DevOps, cloud and platform engineering, Ayoub has developed a perspective that challenges a common misconception about reliability.

Many people associate reliability with incident response. In reality, he sees it as the result of preparation.

Well-designed logging, meaningful error handling and strong observability can dramatically reduce incident resolution time and improve system resilience.

In other words, the ability to handle incidents efficiently is often determined long before an incident ever happens.

When systems provide meaningful information about their behaviour, teams can identify issues faster, understand root causes more easily and make better decisions under pressure. Without that visibility, even relatively small problems can become difficult to diagnose.

This shift in mindset moves reliability away from reactive firefighting and towards proactive engineering. Rather than focusing only on recovery, Site Reliability Engineers help create the conditions that make recovery possible.

The biggest challenge: visibility across complex systems

As technology landscapes continue to evolve, systems rarely exist in isolation. Applications, services, platforms and infrastructure all interact with one another, creating increasingly distributed environments.

For Ayoub, this complexity is not necessarily the biggest problem.

The biggest source of friction is usually the lack of visibility across distributed systems.

When teams cannot clearly see what is happening, diagnosing issues becomes significantly harder.

Missing telemetry, inconsistent monitoring or unclear ownership make incidents much harder to diagnose and resolve.

This is why observability plays such a central role in Site Reliability Engineering. Standardised monitoring, clear operational processes and automation create a shared understanding of system health and behaviour.

The goal is not simply to collect more data. It is to ensure that the right information is available when decisions need to be made. As systems grow more complex, visibility becomes one of the most valuable tools engineering teams have.

Why AI makes reliability even more important

Like many technology disciplines, Site Reliability Engineering is already being influenced by AI. Ayoub sees this impact in several areas. AI is helping teams accelerate infrastructure creation, streamline workflows, support incident investigations and, in certain scenarios, contribute to automated remediation.

But the biggest change may come indirectly:

As AI accelerates software development, teams can deliver features faster than ever.

The ability to ship software more quickly creates enormous opportunities. At the same time, it can also increase operational complexity: more services, more deployments and more dependencies create additional challenges that need to be managed effectively.

That makes the role of SRE even more important, ensuring systems remain reliable, observable and scalable while helping development teams maintain high operational standards as complexity grows.

For Ayoub, reliability and speed are not competing priorities. They need to evolve together, particularly as development accelerates, the foundations that support reliable operations become even more critical.

Automation can help, but people still matter

Automation has become an essential part of modern engineering practices. It reduces repetitive work, improves consistency and helps teams scale processes more effectively.

Yet Ayoub believes automation should be viewed as an enabler rather than a replacement for human expertise. Creating reliable systems at scale starts with thoughtful engineering decisions. Testing strategies, deployment processes, rollback plans and monitoring mechanisms all play a role in building resilience from the beginning.

Automation helps apply these practices consistently, but some decisions still require experience, context and judgement.

Automation and standardisation improve consistency, while human judgment remains essential for architectural decisions and handling unexpected situations.

The challenge is finding the right balance. When routine activities are automated and operational standards are well defined, engineers can spend more time solving the complex problems that technology alone cannot address.

One lesson for engineers building at scale

If there is one lesson Ayoub would share with engineers working in increasingly complex environments, it is surprisingly simple. Start with the business need, and technology offers countless possibilities, but every additional component comes with a cost.

Every new service, gateway or dependency increases operational overhead, so challenge whether it’s truly necessary.

Complexity can sometimes solve problems, but complexity for its own sake often creates new ones.

For Ayoub, the most effective systems are not necessarily the most sophisticated. They are the systems that solve the right problem while remaining understandable, maintainable and resilient over time.

The simplest architecture that meets the requirements is often the most reliable and maintainable.

And in a world where technology is moving faster than ever, that principle remains one of the strongest foundations for reliability.