Software Architecture in Practice: Chapter#4

Technology does not always rhyme with perfection and reliability. Far from it in reality!

— Jean-Michel Jarre

First of all, we need to define a few concepts.

• Availability is defined as the property of software being ready to perform its task when needed. It is a broad concept that includes: 1. Reliability: The consistency of performance, plus considerations like maintenance downtime. 2. Recovery: The ability of a system to repair itself after a break. 3. Fault Masking: Ensuring faults don’t lead to failures, keeping total outage time within required limits.

Essentially, it’s about the system being there and functional whenever it’s required.

• Failure: A failure occurs when a system deviates from its specification and no longer delivers a service consistent with that specification. Crucially, this deviation must be externally visible and observable by the system’s actors or an external observer in the environment.
• Fault: A fault is the underlying cause of a failure. It can be either internal or external to the system and has the potential to cause the system to fail. Through various actions (prevention, tolerance, removal, etc.), a system can become resilient to faults so they do not become visible failures.
• Error: An error is the intermediate state that occurs between the initial occurrence of a fault and the eventual occurrence of a failure (internal bad state).

Now, why do we need to know the difference between Error and Failure?

Because it helps us design strategies to keep the system running, as mentioned, an application can experience an internal fault or error, but as long as the system is able to recover from or mask that issue before it becomes an externally visible deviation from its specification, no failure has occurred.

And because availability is directly tied to the service being ready and functional from the user’s perspective, an application that successfully masks its internal errors without disrupting the user’s experience is still considered available. In other words, you can definitely have faults and errors happening behind the scenes, but as long as the needed functionality is provided without any observable issues, you have no failures, and your system is available.

One of the most demanding tasks in building a high-availability fault-tolerant system is to understand the nature of the failures that can arise during operation. Once those are understood, mitigation strategies can be designed into the system.

Since a system failure is observable by users, the time to repair is the time until the failure is no longer observable.

The notion of “observability” is critical here: If a failure could have been observed, then it is a failure, whether or not it was actually observed by the users.

Availability = MTBF/(MTBF + MTTR)

• MTBF is the Mean Time Between Failures
• MTTR is the Mean Time To Repair

From this formula, it is possible to calculate probabilities and make claims like “the system exhibits 99.999 percent availability”, which means your system can only experience about 5 minutes and 15 seconds of total downtime per year, which is referred to as high availability.

So what determines the availability here is :

• What will cause your system to fail,
• The likelihood of those faults occurring
• How quickly can the system recover?

A critical nuance in calculating these metrics is the treatment of scheduled downtime. When a system is intentionally taken offline for maintenance, it is deemed “not needed” during that window, and thus, this time is completely excluded from the availability calculation, which is often encoded in a service level agreement (SLA).

The detected faults can be categorised prior to being reported and repaired. This categorisation is commonly based on the fault’s severity (critical, major, or minor) and service impact (service-affecting or non-service-affecting).

Availability General Scenario:

• Stimulus Source: Where the fault originates (internal system or external actor).
• Stimulus: The fault or event that could cause system failure.
• Artifact: The part of the system affected (entire system or specific component).
• Environment: The system’s state when the fault occurs (normal, overloaded, degraded, etc.).
• Response: How the system reacts to handle the fault (detect, recover, isolate, continue service).
• Response Measure: How success is evaluated (e.g., uptime %, MTBF, MTTR, downtime limits).

Example: If a database server crashes (stimulus) during peak traffic (environment), the system (Artifact) detects the failure and fails over to a replica (response), ensuring 99.99% uptime (response measure).

Tactics for availability :

• A failure occurs when the system no longer delivers a service consistent with its specification, and this failure is observable to the system’s actors.
• A fault (or combination of faults) can cause a failure. Availability tactics, in turn, are designed to enable a system to prevent or endure system faults so that a service being delivered by the system remains compliant with its specification.

Availability tactics have one of three purposes: Fault detection, Fault recovery, OR Fault prevention.

Detect Faults

Before a system can take action against a fault, it must detect it.

Tactics in this category include:

• Monitor: A component used to monitor the health of other parts of the system, such as processors or memory. It can detect failure or congestion in the network or other shared resources, such as from a denial-of-service attack. It orchestrates software using other tactics in this category to detect malfunctioning components. For example, it can initiate self-tests OR be the component that detects faulty timestamps OR missed heartbeats.
• Ping/echo: A periodic asynchronous request/response message pair exchanged between nodes interconnected via Internet Protocol (IP); it is used to determine reachability and the round-trip delay through the associated network path and is often sent by a system monitor and requires a time threshold to tell the pinging component how long to wait for the echo before considering the pinged component to have failed.
• Heartbeat: a periodic message exchange between a system monitor and a process being monitored. An example of this is the watchdog timer.

A watchdog timer is a hardware or software mechanism that monitors whether a process is operating correctly. If the monitored component fails to signal (“kick,” “feed,” or “reset”) the watchdog within a specified time interval, the watchdog assumes it has become unresponsive and automatically performs a predefined recovery action, such as restarting the process, rebooting the system, or triggering an alert.

The difference between heartbeat and ping/echo lies in who holds the responsibility for initiating the health check — the monitor (Ping/echo) or the component itself (Heartbeat).

• Timestamp: Used to detect incorrect sequences of events, typically in distributed message-passing systems.
• Condition monitoring: This tactic focuses on checking specific conditions within a process or device or validating assumptions that were made during the system’s design.
• Sanity checking: While condition monitoring looks at internal states and design assumptions, sanity checking specifically evaluates the validity or reasonableness of a component’s specific inputs, outputs, or operation results.
• Voting: Comparing computational results from multiple sources (which can use identical replication, functional redundancy, or analytic redundancy) and deciding which result to use if they differ. Typical schemes include the following: 1. Replication redundancy: is the simplest form of voting; here, the components are exact clones of each other, but this can not handle common-mode failure. 2. Functional redundancy: This introduces diversity by having multiple implementations for the same functionality, so it helps eliminate common-mode failure in which all the replicas fail due to an implementation issue, but this can not handle specification errors. 3. Analytic redundancy: permits not only diversity among components’ private sides, but also diversity among the components’ inputs and outputs and was made to tolerate specification errors by using separate requirement specifications for every replica.
• Exception detection: Catching system conditions that alter the normal flow of execution, utilising techniques like 1. System exceptions: They include faults such as divide by zero, bus and address faults, illegal program instructions, and so forth. 2. Timeouts: is a tactic that raises an exception when a component detects that it or another component has failed to meet its timing constraints. 3. Parameter typing: This tactic employs a base class that defines functions to add, find, and iterate over type-length-value (TLV) formatted message parameters. Derived classes then use these base class functions to build and parse messages. **The main goal of parameter typing is to ensure that the sender and the receiver of messages agree on the exact type of the content being exchanged, and to detect any cases where they do not match

4. Parameter fence tactic: **incorporates a known data pattern (such as 0xDEADBEEF) placed immediately after any variable-length parameters of an object.

• Self-test: Components run procedures to test themselves for correct operation by themselves OR by the system monitor.

Recover from Faults

Recovery from faults tactics are preparation, repair tactics and reintroduction tactics.

Firstly, the Preparation and repair tactics:

• Redundant spare: Using duplicate backup components (hot, warm, or cold spares) that can take over the workload if the primary active component fails.
• Rollback: Reverting the system to a previous, known-good state (a checkpoint) after a failure is detected, allowing execution to continue safely.
• Exception handling: Dealing with an exception gracefully (e.g., using error codes or exception classes to mask or repair the fault) rather than letting the system crash.
• Software upgrade: Applying in-service upgrades or patches to executable code without causing downtime or negatively affecting the service and strategies can be as follows:

1. Function patch: Replaces a procedural function in pre-allocated memory while reusing the original entry and exit points.
2. Class patch: Uses a built-in backdoor in object-oriented code to dynamically add data and functions at runtime.
3. Hitless ISSU: Uses redundant backup components to upgrade software and schemas with zero service interruption.

• Retry: Re-attempting an operation up to a certain limit under the assumption that the fault might be temporary (transient).
• Ignore faulty behaviour: It is a tactic where a system intentionally disregards messages sent from a particular source once those messages are determined to be spurious or incorrect.
• Graceful degradation is a recovery and safety tactic designed to maintain the most critical system functions in the presence of component failures, while intentionally dropping or replacing less critical functions.
• Reconfiguration is a fault recovery tactic that attempts to recover from failures by reassigning responsibilities to the (potentially restricted) resources or components left functioning, while maintaining as much functionality as possible.

Secondly, Reintroduction tactics:

• Shadow: This tactic refers to operating a previously failed or in-service-upgraded component in a “shadow mode” for a predefined duration of time prior to reverting the component to an active role. During this duration, its behaviour can be monitored for correctness, and it can repopulate its state incrementally.
• State resynchronisation: This reintroduction tactic is a partner to the redundant spare tactic. When used with
• Active redundancy(Hot): the state resynchronisation occurs organically, since the active and standby components each receive and process identical inputs in parallel. In practice, the states of the active and standby components are periodically compared to ensure synchronisation.
• Passive redundancy(Warm): State resynchronisation is based solely on periodic state information transmitted from the active component(s) to the standby component(s), typically via checkpointing.
• Escalating restart: Recovers from faults by restarting components at progressively larger levels of granularity (ranging from just child threads up to a full executable reload) to minimise service disruption and support graceful degradation.
• Nonstop forwarding: Separates a system’s architecture into a “control plane” (the supervisor making decisions) and a “data plane” (the worker processing data). If the control plane fails, the data plane continues processing along known paths without interruption while the control plane gracefully restarts and rebuilds its state in the background

Prevent Faults

• Removal from service (aka software rejuvenation/therapeutic reboot): This tactic refers to temporarily placing a system component in an out-of-service state for the purpose of mitigating potential system failures.
• Transactions: Systems targeting high-availability services leverage transactional semantics to ensure that asynchronous messages exchanged between distributed components are atomic, consistent, isolated, and durable — properties collectively referred to as the “ACID properties.” The most common realisation of the transactions tactic is the “two-phase commit” (2PC) protocol. This tactic prevents race conditions caused by two processes attempting to update the same data item at the same time.
• Predictive model: Uses a system monitor to track performance metrics and health, allowing the system to predict faults and take corrective action before a critical threshold is reached.
• Exception prevention: Employs safe coding techniques — such as smart pointers, wrappers, and error-correcting codes — to proactively stop system exceptions from ever happening.
• Increase competence set: Expands the range of conditions a component knows how to handle so that it can gracefully deal with unusual states (like a blocked resource) as part of its normal operation, rather than throwing an error

Patterns for Availability

Here we focus on architectural patterns designed to keep a system operational when faults occur.

Redundant Spare Patterns: These patterns rely on having duplicate backup components ready to take over if the primary component fails. They differ primarily in how closely the backup’s state matches the active component’s state:

• Active redundancy (hot spare): All nodes process identical inputs in parallel to maintain a perfectly synchronous state. The spare can take over in milliseconds, but this carries the highest cost and complexity.
• Passive redundancy (warm spare): Only the active nodes process input traffic, but they provide the spare with periodic state updates. This creates a “loosely coupled” state that strikes a practical balance between recovery time and runtime cost.
• Spare (cold spare): Redundant spares remain completely out of service until a failover occurs, requiring a power-on-reset procedure. It is cheaper but has poor recovery performance (a high mean time to repair).
• The core tradeoff among these three is balancing the time to recover from a failure versus the runtime cost required to keep a spare up-to-date.

Triple Modular Redundancy (TMR): This pattern employs three identical components that receive the exact same inputs and perform the exact same tasks. Their outputs are sent to a voting logic mechanism that detects any inconsistencies and decides which result to use, typically by letting the majority rule. It is simple and highly reliable, though it significantly raises component costs.

Circuit Breaker: When a service fails, a calling component might get stuck in an endless loop of retrying to reach it. A circuit breaker steps in to break this endless retry cycle once it determines a fault is happening. This protects the invoking component from becoming unresponsive and prevents the failure from cascading across the rest of the system. However, care must be taken to choose the right timeout values so the circuit breaker doesn’t trip unnecessarily or add too much latency.

Process Pairs: This pattern uses checkpointing and rollback. A backup component continually tracks the state and, if a failure occurs, is ready to roll back to a known safe state to take over operations.

Forward Error Recovery: Instead of falling back to a previous state when an error occurs, this pattern uses built-in error correction (like data redundancy) to fix the error and move the system forward into a new, safe, and possibly degraded state.

At the end, Let’s use the perfect real-life example for this: a modern, self-driving car.

The entire goal of the Availability chapter is to make sure a system is ready to carry out its tasks when you need it, and that it can safely handle faults without causing a complete failure.

Here is how all the major tactics and patterns from the chapter apply to keeping our self-driving car on the road safely:

1. Detect Faults (Figuring out something is wrong). Before the car can fix a problem, it has to know a problem exists.

• Heartbeat / Ping: The car’s main brain constantly sends a periodic “pulse” to the braking system. If the brakes don’t echo back immediately, the car knows the connection has failed.
• Voting (Triple Modular Redundancy Pattern): The car has three separate cameras looking at the traffic light. If two see a red light, but one glitchy camera sees a green light, the system uses “majority rules” to ignore the glitch and stop the car.
• Sanity Checking: If the speedometer suddenly claims the car went from 30 mph to 500 mph in one second, the system flags it as a fault because that is physically impossible based on the car’s state.

2. Prevent Faults (Stopping problems before they happen) It is better to avoid a crash entirely than to recover from one.

• Predictive Model: The car monitors its engine temperature, battery levels, and internal queues. It notices the temperature is creeping up and predicts an overheat, so it automatically opens cooling vents before the engine actually reaches a critical, failing threshold.
• Removal from Service (Therapeutic Reboot): Every night when the car is parked in your garage, it automatically reboots its navigation and entertainment software to scrub away memory leaks and hidden bugs before they can cause a freeze-up the next day.
• Increase Competence Set: If the car’s software requests access to a locked GPS map file, instead of crashing and throwing a fatal exception, it is programmed to simply wait and try again later.

3. Recover from Faults: Preparation and Repair (Enduring the hit) When something does break, the car needs to keep you safe.

• Graceful Degradation: You drive into a long underground tunnel, and the car loses its GPS signal. Instead of the whole driving system shutting down, it gracefully degrades its service by falling back to “dead reckoning” (using the steering wheel angle and speedometer to guess its location) until it sees the sky again.
• Ignore Faulty Behaviour: You drive through a puddle and a parking sensor gets covered in mud, causing it to scream that you are about to hit a wall. The car’s computer realises the data is spurious and simply ignores that specific sensor.
• Circuit Breaker Pattern: The car’s computer is trying to connect to a broken satellite radio server. Instead of getting stuck in an endless loop of retrying (which could freeze the whole dashboard), the circuit breaker “trips” and stops the retries, protecting the rest of the car’s systems.

4. Recover from Faults: Reintroduction (Bringing the fixed part back in). Once a part is fixed or rebooted, it must be eased back into the driving system without jerking the steering wheel.

• Escalating Restart: The music app freezes. The car does not reboot the entire vehicle while you are driving at 70 mph. It tries the smallest restart first (just the app). If that fails, it restarts the dashboard screen (a larger level), leaving the engine and steering completely untouched.
• Redundant Spare & State Resynchronization: The car has a backup “warm spare” steering computer. If the main steering computer fails, the backup takes over. But before it can safely drive, it uses state resynchronization to copy the exact current speed, direction, and wheel angle from the active components so the transition is perfectly smooth.

The Summary: By combining these tactics, the car ensures that its Mean Time Between Failures (MTBF) is very high, and its Mean Time To Repair (MTTR) is so fast that the passenger never even notices a glitch. This is the essence of building a highly available architecture!