Enhancing Resilience in Distributed Systems with the Circuit Breaker design pattern

In modern distributed systems, components and services often depend on each other to function correctly. The Circuit Breaker pattern is a design pattern that helps protect systems from cascading failures by monitoring the health of a dependent system and, if it detects that the system is failing, it will stop sending requests to that system. This prevents the failure from spreading to other systems in the system. It is particularly useful in microservices architectures, where each service is a separate, independent unit. If one service fails, the Circuit Breaker will prevent requests from being sent to the failed service, and the other services will continue to operate normally.

How does the circuit breaker pattern work?

The circuit breaker pattern works by monitoring the number of failures that occur when a request is made to a dependent system. If the number of failures exceeds a certain threshold, the circuit breaker will open. This means that the circuit breaker will stop sending requests to the dependent system.

Once the circuit breaker is open, it will remain open for a certain amount of time. During this time, the circuit breaker will continue to monitor the dependent system. If the number of failures decreases below the threshold, the circuit breaker will close. This means that the circuit breaker will start sending requests to the dependent system again.

When should you use the circuit breaker pattern?

The circuit breaker pattern should be used whenever you have a system that depends on another system. This is especially important in microservices architectures, where each service is a separate, independent unit.

The circuit breaker pattern can help to protect your system from cascading failures. If one service fails, the circuit breaker will prevent the failure from spreading to other services. This can help to keep your system up and running even when there are failures in individual services.

Circuit Breaker States

A Circuit Breaker can be in one of three states:

1. Closed: The Circuit Breaker allows all function calls to pass through. If failures reach a specified threshold, the Circuit Breaker transitions to the Open state.
2. Open: The Circuit Breaker blocks all function calls and returns an error. After a predefined timeout, it transitions to the Half-Open state.
3. Half-Open: The Circuit Breaker allows a limited number of requests to pass through. If these requests succeed, it transitions back to the Closed state. If they fail, it returns to the Open state.

Implementation

Java

import java.util.concurrent.TimeUnit;

public class CircuitBreaker {

    private boolean open = false;
    private int failureCount = 0;
    private int threshold = 5;
    private TimeUnit timeUnit = TimeUnit.SECONDS;

    public void tryCall(Callable<Object> callable) throws Exception {
        if (open) {
            throw new RuntimeException("Circuit breaker is open");
        }

        try {
            callable.call();
        } catch (Exception e) {
            failureCount++;
            if (failureCount >= threshold) {
                open = true;
                TimeUnit.sleep(timeUnit.toMillis(10));
            }
        }
    }

    public void close() {
        open = false;
        failureCount = 0;
    }
}

Explanation

This Java code defines a CircuitBreaker class that is used to protect a system from cascading failures. When a certain number of failures occur within a specified time frame, the circuit breaker “opens” and prevents further calls to the potentially failing operation. The circuit breaker can be manually “closed” to allow calls again.

Let’s break down the code:

1. Import statement:
   import java.util.concurrent.TimeUnit;
   This imports the TimeUnit class from the java.util.concurrent package. The TimeUnit class provides a way to work with time durations at a given unit of granularity (e.g., seconds, minutes).
2. Class definition:
   public class CircuitBreaker { ... }
   This defines the CircuitBreaker class with instance variables and methods.
3. Instance variables:

• private boolean open = false; – A boolean variable that represents the state of the circuit breaker (open or closed). By default, it’s set to false (closed).
• private int failureCount = 0; – An integer variable that counts the number of failures encountered.
• private int threshold = 5; – An integer variable that sets the threshold for the number of failures allowed before the circuit breaker opens.
• private TimeUnit timeUnit = TimeUnit.SECONDS; – A TimeUnit variable that represents the time unit used for the circuit breaker’s sleep duration.

1. tryCall method:
   public void tryCall(Callable<Object> callable) throws Exception { ... }
   This method takes a Callable<Object> as input and tries to call it. If the circuit breaker is open, it throws a RuntimeException. If the call fails with an exception, it increments the failureCount. If the failureCount reaches the threshold, the circuit breaker opens, and the system sleeps for 10 seconds before allowing any more calls.
2. close method:
   public void close() { ... }
   This method closes the circuit breaker and resets the failureCount to 0. This allows the system to try calls again.

In summary, the CircuitBreaker class provides a simple mechanism to protect a system from cascading failures by stopping calls to a potentially failing operation when a certain number of failures have occurred within a specified time frame.

Circuit Breaker Libraries

JVM Languages

1. Hystrix (Java): Hystrix is a popular library developed by Netflix for latency and fault tolerance. It provides a circuit breaker implementation and additional features like thread and semaphore isolation and fallbacks. However, Hystrix is no longer in active development, and Netflix recommends using Resilience4j for new projects. GitHub: https://github.com/Netflix/Hystrix
2. Resilience4j (Java): Resilience4j is a lightweight, modular fault tolerance library for Java 8 and higher, inspired by Netflix Hystrix. It provides a circuit breaker implementation, along with other resilience patterns like rate limiting, retries, and bulkhead isolation. GitHub: https://github.com/resilience4j/resilience4j
3. Akka Circuit Breaker (Scala): The Akka framework for building reactive systems in Scala and Java includes a circuit breaker implementation as part of its Akka Actor library. It is designed for integration with the actor model and asynchronous programming. Documentation: https://doc.akka.io/docs/akka/current/common/circuitbreaker.html
   

Python

1. PyBreaker: PyBreaker is a simple implementation of the Circuit Breaker pattern for Python applications. It provides a Circuit Breaker class that can be used to wrap any function or method and comes with pluggable storage backends for the breaker’s state. GitHub: https://github.com/danielfm/pybreaker
2. CircuitPython: CircuitPython is another lightweight library that implements the Circuit Breaker pattern for Python applications. It provides decorators that can be applied to functions or methods and comes with additional features like monitoring and logging. GitHub: https://github.com/numberoverzero/circuit
3. Tenacity: Tenacity is a general-purpose retry library for Python that can also be used to implement a circuit breaker pattern. It is an Apache 2.0 licensed library that can be used with both synchronous and asynchronous code. GitHub: https://github.com/jd/tenacity

Javascript

1. Opossum: Opossum is a circuit breaker library for Node.js that provides a simple and easy-to-use API. It can be used with any JavaScript function and has built-in support for Promises. It also includes features like timeouts, error filtering, and fallback functions. GitHub: https://github.com/nodeshift/opossum
2. Brakes: Brakes is another circuit breaker library for Node.js that provides an easy-to-use and extensible API. It comes with support for Promises, timeouts, and fallback functions. Brakes also include built-in stats and event reporting features for monitoring and observability. GitHub: https://github.com/awolden/brakes
3. Cockpit: Cockpit is a lightweight circuit breaker library for JavaScript that can be used in both Node.js and browser environments. It provides a simple implementation of the circuit breaker pattern with support for timeouts and fallback functions. GitHub: https://github.com/creately/cockpit
4. Circuit-Breaker-JS: Circuit-Breaker-JS is another simple circuit breaker implementation for JavaScript applications that can be used in both Node.js and browser environments. It supports timeouts, fallbacks, and custom state storage backends. GitHub: https://github.com/yammer/circuit-breaker-js

These libraries provide various levels of sophistication and customization for implementing circuit breakers in JVM languages, Python and Javascript. Choose the one that best fits your project requirements and the language you are using.