What Is Chaos Engineering? | Vantage

Chaos Engineering Chaos engineering is the practice of deliberately introducing controlled failures into a production system to discover weaknesses before they cause unplanned outages. Pioneered by Netflix with their Chaos Monkey tool (which randomly terminates production instances), chaos engineering validates that a system is resilient to the failures you anticipate and surfaces failure modes you did not anticipate. Experiments follow a scientific process: define a steady state, form a hypothesis about how the system will behave during a failure, inject the failure, and observe whether the steady state holds.

Why chaos engineering matters

Distributed systems fail in complex ways that are impossible to predict from code review and unit tests alone. A service that handles database failover correctly in a staging test may fail when the failover happens during peak load with an active cache refresh running simultaneously. Chaos engineering is the only way to discover these compound failure modes before users experience them. Netflix found that running Chaos Monkey in production actually improved reliability: teams built more resilient services because they knew the monkey was coming.

How it works

Start with a steady state definition — the metrics that indicate normal operation (error rate, p99 latency, availability). Form a hypothesis: if we terminate one instance of service X, the system will continue serving requests within SLO. Run the experiment during business hours when the team is available to respond. Observe the metrics. If the steady state holds, confidence in resilience increases. If it does not, you have found a weakness to fix. Gradually expand scope: single instance failure, network partition, database slowdown, dependency timeout. Use tools like Gremlin, AWS Fault Injection Simulator, or Netflix Chaos Monkey.

Common mistakes

• Running chaos experiments without a steady state definition — you cannot measure resilience without defining normal

• Starting in production before validating in staging — chaos engineering should mature through environments before hitting production

• No game day process — chaos experiments without an incident response team standing by are high-risk drills

• Treating chaos engineering as a one-time event — resilience degrades as systems change; experiments must be re-run as the system evolves

• Skipping the hypothesis — chaos engineering without a hypothesis is random destruction, not science