Distributed Systems

Overview

Resilient Circuit is designed for distributed systems where multiple service instances need to coordinate failure handling. Using PostgreSQL as a shared storage backend, circuit breakers can synchronize state across all instances of your application.

Why Distributed Circuit Breakers?

In a distributed architecture, you want:

Coordinated Failure Handling: When one instance detects a failing external service, all instances should stop calling it immediately to prevent cascading failures.
Shared State: Circuit breaker state (OPEN, CLOSED, HALF_OPEN) should be consistent across all instances.
Persistence: State should survive instance restarts and deployments.
Atomic Operations: State transitions must be thread-safe and race-condition-free.

Architecture Patterns

Single Service, Multiple Instances

The most common pattern: multiple instances of the same service share circuit breaker state.

# Instance 1 (Pod/Container A)
from resilient_circuit import CircuitProtectorPolicy

payment_cb = CircuitProtectorPolicy(
    resource_key="stripe_payment_api",  # Shared key
    cooldown=timedelta(minutes=5)
)

# Instance 2 (Pod/Container B)
payment_cb = CircuitProtectorPolicy(
    resource_key="stripe_payment_api",  # Same key
    cooldown=timedelta(minutes=5)
)

Flow:

Instance 1 calls Stripe API → fails repeatedly
Instance 1’s circuit breaker opens → state saved to PostgreSQL
Instance 2 immediately sees OPEN state from PostgreSQL
Instance 2 stops calling Stripe API → prevents further failures

Multiple Services, Shared Resource

Different services calling the same external resource:

# Payment Service
payment_cb = CircuitProtectorPolicy(
    resource_key="external_fraud_check_api",
    cooldown=timedelta(minutes=2)
)

# Order Service
order_cb = CircuitProtectorPolicy(
    resource_key="external_fraud_check_api",  # Same resource
    cooldown=timedelta(minutes=2)
)

# User Service
user_cb = CircuitProtectorPolicy(
    resource_key="external_fraud_check_api",  # Same resource
    cooldown=timedelta(minutes=2)
)

When any service detects fraud check API failures, all services stop calling it.

Per-Instance vs Shared Keys

You can mix local and shared circuit breakers:

# Local circuit breaker (per-instance)
local_cb = CircuitProtectorPolicy(
    resource_key=f"local_cache_{instance_id}",  # Unique per instance
    storage=InMemoryStorage()  # In-memory only
)

# Shared circuit breaker (distributed)
shared_cb = CircuitProtectorPolicy(
    resource_key="shared_external_api",  # Same across instances
    # Automatically uses PostgreSQL if configured
)

Use Cases

Microservices Architecture

Scenario: E-commerce platform with multiple microservices

# All services configure the same circuit breakers
shared_resources = {
    "payment_gateway": CircuitProtectorPolicy(
        resource_key="payment_gateway",
        cooldown=timedelta(minutes=5),
        failure_limit=Fraction(3, 10)
    ),
    "inventory_service": CircuitProtectorPolicy(
        resource_key="inventory_service",
        cooldown=timedelta(seconds=30),
        failure_limit=Fraction(2, 5)
    ),
    "shipping_api": CircuitProtectorPolicy(
        resource_key="shipping_api",
        cooldown=timedelta(minutes=2),
        failure_limit=Fraction(1, 3)
    ),
}

# In Order Service
@shared_resources["payment_gateway"]
def charge_payment(order):
    # Payment processing
    pass

# In Fulfillment Service
@shared_resources["payment_gateway"]
def refund_payment(order):
    # Refund processing
    pass

Both services benefit from shared circuit breaker state.

Kubernetes Deployments

Scenario: Kubernetes deployment with auto-scaling

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 5  # 5 instances
  template:
    spec:
      containers:
      - name: payment-service
        image: payment-service:latest
        env:
        - name: RC_DB_HOST
          value: postgresql.default.svc.cluster.local
        - name: RC_DB_NAME
          value: resilient_circuit_db
        - name: RC_DB_USER
          valueFrom:
            secretKeyRef:
              name: postgres-credentials
              key: username
        - name: RC_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-credentials
              key: password

# payment_service/main.py
from resilient_circuit import CircuitProtectorPolicy

# All 5 replicas share this circuit breaker state
external_api_cb = CircuitProtectorPolicy(
    resource_key="external_kyc_api",
    cooldown=timedelta(minutes=10)
)

When Kubernetes scales to 10 replicas, all instances automatically share state.

Serverless Functions

Scenario: AWS Lambda or Google Cloud Functions

import os
from resilient_circuit import CircuitProtectorPolicy

# Lambda function
def lambda_handler(event, context):
    # Circuit breaker persists across invocations via PostgreSQL
    cb = CircuitProtectorPolicy(
        resource_key="third_party_api",
        cooldown=timedelta(minutes=5)
    )

    @cb
    def call_third_party():
        # API call
        pass

    result = call_third_party()
    return result

Multiple Lambda invocations share the same circuit breaker state through PostgreSQL.

Multi-Region Deployments

Scenario: Services deployed across multiple regions

# us-east-1 region
cb_east = CircuitProtectorPolicy(
    resource_key="global_payment_processor",
    # Connected to PostgreSQL primary in us-east-1
)

# eu-west-1 region
cb_eu = CircuitProtectorPolicy(
    resource_key="global_payment_processor",
    # Connected to PostgreSQL replica in eu-west-1
)

Note: For multi-region setups, consider PostgreSQL replication latency and eventual consistency.

State Synchronization

How It Works

State Read: Circuit breaker loads current state from PostgreSQL on initialization
State Update: Every state change is immediately persisted to PostgreSQL
Atomic Operations: PostgreSQL row-level locking ensures atomicity
New Instance: When a new instance starts, it loads the latest state

State Transition Example

# Time T0: All instances see CLOSED state
# Instance 1, 2, 3 → CLOSED

# Time T1: Instance 1 experiences failures
# Instance 1 → Executes 10 calls, 6 fail
# Instance 1 → Transitions to OPEN
# Instance 1 → Saves state to PostgreSQL

# Time T2: Instance 2 checks state before next call
# Instance 2 → Loads state from PostgreSQL
# Instance 2 → Sees OPEN state
# Instance 2 → Blocks call without executing

# Time T3: After cooldown expires
# Instance 3 → First to attempt call after cooldown
# Instance 3 → Transitions to HALF_OPEN
# Instance 3 → Saves state to PostgreSQL
# Instance 3 → Attempts call

# Time T4: Call succeeds
# Instance 3 → Transitions to CLOSED
# Instance 3 → Saves state to PostgreSQL
# All instances → See CLOSED state on next check

Concurrency Handling

PostgreSQL’s FOR UPDATE locks prevent race conditions:

-- Internal query used by Resilient Circuit
SELECT state, failure_count, open_until
FROM rc_circuit_breakers
WHERE resource_key = 'payment_api'
FOR UPDATE;  -- Locks the row

This ensures only one instance can transition state at a time.

Performance Optimization

Circuit Breaker Caching

Reuse circuit breaker instances:

# Good: Create once, reuse many times
payment_cb = CircuitProtectorPolicy(
    resource_key="payment_service"
)

@payment_cb
def process_payment():
    pass

# Call multiple times with same instance
for order in orders:
    process_payment()

# Bad: Creating new instance on every call
def process_payment():
    cb = CircuitProtectorPolicy(
        resource_key="payment_service"
    )
    # ...

Lazy Loading

Load state from PostgreSQL only when needed:

from resilient_circuit import CircuitProtectorPolicy
from resilient_circuit.storage import create_storage

# Initialize storage once
storage = create_storage()

# Share storage across circuit breakers
cb1 = CircuitProtectorPolicy(
    resource_key="api_1",
    storage=storage
)

cb2 = CircuitProtectorPolicy(
    resource_key="api_2",
    storage=storage
)

Connection Pooling

For high-throughput applications, consider PostgreSQL connection pooling:

# Using pgbouncer or similar
# Set RC_DB_HOST to pgbouncer address
export RC_DB_HOST=pgbouncer.internal.svc.cluster.local
export RC_DB_PORT=6432

Monitoring and Observability

Real-Time Monitoring

Query circuit breaker state across all instances:

-- Dashboard query: Current state of all circuit breakers
SELECT
    resource_key,
    state,
    failure_count,
    CASE
        WHEN state = 'OPEN' THEN
            EXTRACT(EPOCH FROM (open_until - NOW()))
        ELSE 0
    END as seconds_until_halfopen,
    updated_at
FROM rc_circuit_breakers
ORDER BY updated_at DESC;

Alerting

Set up alerts for critical circuit breakers:

-- Alert: Payment service circuit breaker opened
SELECT resource_key, state, updated_at
FROM rc_circuit_breakers
WHERE resource_key = 'payment_service'
  AND state = 'OPEN'
  AND updated_at > NOW() - INTERVAL '5 minutes';

Metrics Collection

Collect metrics for monitoring systems:

from resilient_circuit import CircuitProtectorPolicy

def status_change_callback(policy, old_state, new_state):
    # Send metric to Prometheus/DataDog/etc.
    metrics.increment(
        'circuit_breaker.state_change',
        tags=[
            f'resource:{policy.resource_key}',
            f'old_state:{old_state.value}',
            f'new_state:{new_state.value}'
        ]
    )

cb = CircuitProtectorPolicy(
    resource_key="critical_api",
    on_status_change=status_change_callback
)

Best Practices

Use Consistent Resource Keys

Ensure all instances use identical resource_key values.
Configure Appropriate Timeouts

Set cooldown periods that match your service’s recovery time.
Test Failover Scenarios

Verify circuit breakers work correctly during:
- PostgreSQL downtime (should fallback to in-memory)
- Network partitions
- Instance scaling events
Monitor Database Performance

Circuit breakers add database load. Monitor:
- Connection count
- Query latency
- Lock contention
Plan for PostgreSQL Maintenance

Circuit breakers gracefully degrade to in-memory storage if PostgreSQL is unavailable.

Troubleshooting

Instances Not Syncing

Symptom: Different instances show different circuit states

Solutions:

Verify all instances connect to the same PostgreSQL database
Check resource_key is identical across instances
Ensure network connectivity between instances and database
Check for clock skew across instances

High Database Load

Symptom: PostgreSQL CPU or connection count high

Solutions:

Add connection pooling (pgbouncer)
Increase PostgreSQL connection limits
Optimize indexes (already created by CLI)
Use fewer circuit breakers or longer cache times

State Not Persisting

Symptom: Circuit breaker state resets after instance restart

Solutions:

Verify PostgreSQL storage is configured (check RC_DB_* env vars)
Check circuit breaker is using PostgreSQL, not in-memory storage
Verify database permissions allow writes
Check application logs for storage errors

Distributed Systems

Overview

Why Distributed Circuit Breakers?

Architecture Patterns

Single Service, Multiple Instances

Multiple Services, Shared Resource

Per-Instance vs Shared Keys

Use Cases

Microservices Architecture

Kubernetes Deployments

Serverless Functions

Multi-Region Deployments

State Synchronization

How It Works

State Transition Example

Concurrency Handling

Performance Optimization

Circuit Breaker Caching

Lazy Loading

Connection Pooling

Monitoring and Observability

Real-Time Monitoring

Alerting

Metrics Collection

Best Practices

Troubleshooting

Instances Not Syncing

High Database Load

State Not Persisting

See Also