# Resilience Patterns Implementation

This document describes the resilience patterns implemented for the Amadeus API client to ensure system stability and graceful degradation when external services fail.

## Overview

The B2B Flight Ticketing Platform implements three key resilience patterns for all Amadeus API calls:

1. **Circuit Breaker**: Prevents cascade failures
2. **Retry with Exponential Backoff**: Handles transient failures
3. **Timeout**: Prevents hanging requests

These patterns are implemented using Resilience4j library and are configured in `application.yml`.

## Requirements

- **Requirement 13.1**: Retry with exponential backoff when Amadeus API is unavailable
- **Requirement 13.4**: Circuit breaker to prevent cascade failures

## Circuit Breaker Pattern

### Purpose
Prevents the application from repeatedly trying to execute an operation that's likely to fail, allowing the external service time to recover.

### Configuration
```yaml
resilience4j:
  circuitbreaker:
    instances:
      amadeus:
        slidingWindowSize: 10
        minimumNumberOfCalls: 5
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 3
```

### States

1. **CLOSED** (Normal Operation)
   - All requests pass through to Amadeus API
   - Failures are recorded in sliding window
   - If failure rate exceeds 50% in last 10 calls, circuit opens

2. **OPEN** (Service Degraded)
   - Requests are immediately rejected without calling Amadeus API
   - Fallback method is invoked
   - After 10 seconds, circuit transitions to HALF_OPEN

3. **HALF_OPEN** (Testing Recovery)
   - Limited number of requests (3) are allowed through
   - If requests succeed, circuit closes
   - If requests fail, circuit reopens

### Behavior

When circuit is OPEN:
- Requests fail immediately with `AmadeusServiceException`
- Error type: `CIRCUIT_BREAKER_OPEN`
- Message: "Service is temporarily unavailable due to high error rate"

### Monitoring

Circuit breaker state transitions are logged:
```
WARN: Circuit breaker state transition: CLOSED -> OPEN for amadeus
WARN: Circuit breaker state transition: OPEN -> HALF_OPEN for amadeus
INFO: Circuit breaker state transition: HALF_OPEN -> CLOSED for amadeus
```

## Retry Pattern

### Purpose
Automatically retries failed requests to handle transient network issues or temporary service unavailability.

### Configuration
```yaml
resilience4j:
  retry:
    instances:
      amadeus:
        maxAttempts: 3
        waitDuration: 1s
        exponentialBackoffMultiplier: 2
        maxWaitDuration: 10s
```

### Retry Strategy

**Exponential Backoff Schedule:**
1. Initial attempt: Immediate
2. First retry: Wait 1 second
3. Second retry: Wait 2 seconds (1s × 2)
4. Third retry: Wait 4 seconds (2s × 2)

**Total time for all retries:** Up to ~7 seconds

### Retryable Exceptions

The following exceptions trigger automatic retry:
- `java.net.SocketTimeoutException` - Network timeout
- `org.springframework.web.client.ResourceAccessException` - Connection issues
- `org.springframework.web.client.HttpServerErrorException.ServiceUnavailable` - 503 errors
- `org.springframework.web.client.HttpServerErrorException.GatewayTimeout` - 504 errors

### Non-Retryable Exceptions

The following exceptions do NOT trigger retry:
- `org.springframework.web.client.HttpClientErrorException` - 4xx errors (client errors)
- Authentication failures
- Validation errors

### Behavior

When all retries are exhausted:
- Fallback method is invoked
- `AmadeusServiceException` is thrown
- Error type: `SERVICE_UNAVAILABLE`

### Monitoring

Retry attempts are logged:
```
WARN: Retry attempt 1 for amadeus due to: Connection timeout
WARN: Retry attempt 2 for amadeus due to: Connection timeout
ERROR: All retry attempts exhausted for amadeus: Connection timeout
```

## Timeout Pattern

### Purpose
Limits the execution time of API calls to prevent requests from hanging indefinitely.

### Configuration
```yaml
resilience4j:
  timelimiter:
    instances:
      amadeus:
        timeoutDuration: 10s
        cancelRunningFuture: true
```

### Behavior

- Maximum execution time: 10 seconds
- If timeout is exceeded:
  - Running future is cancelled
  - `TimeoutException` is thrown
  - Fallback method is invoked
  - Error type: `TIMEOUT`

### Monitoring

Timeouts are logged:
```
ERROR: Time limiter timeout for amadeus: execution exceeded timeout duration
```

## Fallback Methods

When all resilience patterns fail (circuit open, retries exhausted, or timeout), fallback methods provide graceful degradation.

### Flight Search Fallback

```java
private AmadeusFlightSearchResponse searchFlightsFallback(
    AmadeusFlightSearchRequest request, 
    Exception ex
)
```

**Behavior:**
- Logs detailed error information
- Determines error type (timeout, circuit breaker, service unavailable)
- Throws `AmadeusServiceException` with appropriate error message
- Does NOT return mock data (fails fast)

### Fare Rules Fallback

```java
private AmadeusFareRulesResponse getFareRulesFallback(
    String offerId, 
    Exception ex
)
```

**Behavior:**
- Logs detailed error information
- Determines error type
- Throws `AmadeusServiceException` with appropriate error message

## Error Types

The `AmadeusServiceException` includes an error type for better error handling:

```java
public enum ErrorType {
    TIMEOUT,                  // Request exceeded 10 seconds
    CIRCUIT_BREAKER_OPEN,     // Too many failures, circuit is open
    SERVICE_UNAVAILABLE,      // Service is down or unreachable
    UNKNOWN                   // Other errors
}
```

## Usage Example

```java
@Service
public class FlightService {
    
    @Autowired
    private AmadeusClient amadeusClient;
    
    public FlightSearchResponse searchFlights(FlightSearchRequest request) {
        try {
            // Resilience patterns are applied automatically
            AmadeusFlightSearchResponse response = amadeusClient.searchFlights(
                convertToAmadeusRequest(request)
            );
            return convertToFlightSearchResponse(response);
            
        } catch (AmadeusServiceException ex) {
            // Handle specific error types
            switch (ex.getErrorType()) {
                case TIMEOUT:
                    log.warn("Flight search timed out");
                    throw new ServiceException("Search is taking longer than expected");
                    
                case CIRCUIT_BREAKER_OPEN:
                    log.error("Amadeus service is degraded");
                    throw new ServiceException("Flight search is temporarily unavailable");
                    
                case SERVICE_UNAVAILABLE:
                    log.error("Amadeus service is down");
                    throw new ServiceException("Unable to search flights at this time");
                    
                default:
                    log.error("Unexpected error", ex);
                    throw new ServiceException("An error occurred while searching flights");
            }
        }
    }
}
```

## Testing Resilience Patterns

### Testing Circuit Breaker

To test circuit breaker behavior:

1. Simulate multiple failures (>50% in 10 calls)
2. Verify circuit opens
3. Verify requests are rejected immediately
4. Wait 10 seconds
5. Verify circuit transitions to HALF_OPEN
6. Send successful requests
7. Verify circuit closes

### Testing Retry

To test retry behavior:

1. Simulate transient failure (e.g., timeout)
2. Verify retry attempts with exponential backoff
3. Verify successful retry stops further attempts
4. Verify exhausted retries invoke fallback

### Testing Timeout

To test timeout behavior:

1. Simulate slow API response (>10 seconds)
2. Verify request is cancelled after 10 seconds
3. Verify TimeoutException is thrown
4. Verify fallback is invoked

## Monitoring and Alerting

### Metrics

The following metrics are exposed via Spring Boot Actuator:

- `resilience4j.circuitbreaker.state` - Current circuit breaker state
- `resilience4j.circuitbreaker.failure.rate` - Failure rate percentage
- `resilience4j.retry.calls` - Number of retry attempts
- `resilience4j.timelimiter.calls` - Number of timeout occurrences

### Health Indicator

Circuit breaker health is exposed at `/actuator/health`:

```json
{
  "status": "UP",
  "components": {
    "circuitBreakers": {
      "status": "UP",
      "details": {
        "amadeus": {
          "status": "UP",
          "state": "CLOSED",
          "failureRate": "0.0%"
        }
      }
    }
  }
}
```

### Recommended Alerts

1. **Circuit Breaker Opens**
   - Alert: CRITICAL
   - Action: Investigate Amadeus API health

2. **High Retry Rate**
   - Alert: WARNING
   - Threshold: >20% of requests require retry
   - Action: Check network connectivity

3. **Frequent Timeouts**
   - Alert: WARNING
   - Threshold: >10% of requests timeout
   - Action: Review timeout configuration or Amadeus API performance

## Configuration Tuning

### Aggressive (Fast Failure)
```yaml
resilience4j:
  circuitbreaker:
    instances:
      amadeus:
        slidingWindowSize: 5
        failureRateThreshold: 30
        waitDurationInOpenState: 5s
  retry:
    instances:
      amadeus:
        maxAttempts: 2
        waitDuration: 500ms
  timelimiter:
    instances:
      amadeus:
        timeoutDuration: 5s
```

### Conservative (More Tolerant)
```yaml
resilience4j:
  circuitbreaker:
    instances:
      amadeus:
        slidingWindowSize: 20
        failureRateThreshold: 70
        waitDurationInOpenState: 30s
  retry:
    instances:
      amadeus:
        maxAttempts: 5
        waitDuration: 2s
  timelimiter:
    instances:
      amadeus:
        timeoutDuration: 30s
```

## Best Practices

1. **Always use fallback methods** - Never let resilience patterns fail silently
2. **Log all failures** - Include context for debugging
3. **Monitor circuit breaker state** - Set up alerts for state transitions
4. **Tune based on SLA** - Adjust thresholds based on Amadeus API SLA
5. **Test failure scenarios** - Regularly test circuit breaker and retry behavior
6. **Use appropriate error types** - Help clients handle errors appropriately
7. **Document expected behavior** - Make it clear what happens when patterns activate

## References

- [Resilience4j Documentation](https://resilience4j.readme.io/)
- [Circuit Breaker Pattern](https://martinfowler.com/bliki/CircuitBreaker.html)
- [Retry Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/retry)
- Requirements: 13.1, 13.4