Building an Observability Culture: Insights from Engineering Practices
Written on
Chapter 1: Understanding the Need for Observability
In the realm of engineering, our desires are straightforward:
- A restful night without disruptions due to production issues
- A seamless operational experience
- Continuous feedback from production to enhance our systems
Observability plays a crucial role in achieving these goals.
Let’s delve into a historical example. During World War II, the British Navy recognized a significant loss of aircraft and understood the need to enhance their planes’ armor for better survival rates. However, the challenge lay in determining the optimal amount of armor; too much would hinder maneuverability, while too little would increase vulnerability.
The navy decided to study the returning aircraft to identify bullet hole patterns.
The analysis revealed that the planes were frequently hit in the wings and fuselage, but not in the cockpit or tail. The assumption was to reinforce the most damaged areas. However, statisticians like Abraham Wald pointed out a critical oversight: the real issue lay in the areas that weren’t returning for analysis—specifically, the engine and fuselage.
Wald illustrated a concept known as Survivorship Bias, where we tend to focus on available data instead of the complete picture. This bias, similar to that faced by the Navy, can lead to misunderstandings in our production systems. Thus, gaining comprehensive insights from these systems is imperative, highlighting the significance of observability.
Section 1.1: Observability vs. Monitoring
A discussion on observability cannot overlook monitoring. Some argue that observability is merely a rebranding of monitoring, while others maintain they are fundamentally different. Regardless of the terminology, the key is to reap the benefits and implement it effectively.
Monitoring serves two primary functions:
- It allows us to respond to predictable failures, such as disk space issues or request delays.
- It aids in testing, ensuring the system's correctness.
Modern software teams excel at managing and testing failures that can be identified through traditional methods. Techniques like retries, auto-scaling, and failovers enhance system resilience. However, as the Navy learned, we can miss crucial data, leading to challenges in identifying the root cause of seemingly unrelated issues. This is where observability becomes essential, particularly for cloud-based systems.
Chapter 2: Achieving Effective Observability
In this video titled "Improving Observability and Testing In Production," you'll learn how to enhance observability in production environments, ensuring better system resilience and recovery.
To achieve observability, we must focus on two key areas:
- Collecting meaningful data enriched with context
- Utilizing effective tools to visualize and analyze this data
Data Collection
Today, effective data collection encompasses logs, metrics, and traces. While telemetry is crucial, it must be designed to yield insights about unknown issues. For instance, if you receive alerts about unusual spikes in failed requests, simply reviewing metrics won’t suffice. You must have a well-structured telemetry system to uncover underlying problems effectively.
Consider these points for better instrumentation:
- Who is accessing your service? (e.g., user details)
- What are they requesting? (e.g., URL paths)
- How did the service respond? (e.g., response times)
- Are business goals being met? (e.g., SLA compliance)
- What is the broader context? (e.g., resource usage)
Effective Tooling
Having gathered useful data, the next step is employing tools that can efficiently merge metrics, logs, and traces for easy debugging. A good tool should be:
- Fast and capable of handling large datasets
- Able to provide in-depth queries
- Facilitate collaboration and documentation
The second video, "Container Observability with AWS | AWS Events," explores how to implement observability within containerized environments using AWS services.
Testing and Recovery in Production
App teams traditionally invest significant effort into testing before production releases. However, gaining confidence in production environments is equally vital. To enhance confidence, consider:
- Robust instrumentation
- Utilizing feature flags for new functionalities
- Implementing canary releases to test changes with selected users
- Establishing efficient rollback mechanisms
Imagine if, upon receiving alerts about failed requests from European users, you quickly trace the issue to an IP blockage caused by a deployment. With effective instrumentation, you can promptly identify and rectify the problem, documenting the necessary root cause analysis.
Assessing Your Readiness
Progressing towards better observability and recovery in production is a journey, and each team will be at different stages. Awareness of your current state and planning for the next steps is crucial.
Observability Radar: Tools and Platforms
The ThoughtWorks tech radar frequently highlights tools that can aid software development teams in achieving observability. A consolidated observability radar can serve as a valuable resource for identifying suitable tools and platforms.
Building an Observability-Driven Culture
As Brian Knox from Digital Ocean states, "The goal of an Observability team is not to collect logs, metrics, or traces. It is to cultivate an engineering culture based on facts and feedback." Teams should integrate observability into their planning processes and use production insights for informed decision-making.
References: