# Booking Modification Monitoring

This directory contains monitoring and observability configurations for the booking modification feature.

## Overview

The booking modification feature includes comprehensive monitoring through:
- **Metrics**: Micrometer metrics exposed via Spring Boot Actuator
- **Alerts**: Prometheus alert rules for critical thresholds
- **Dashboards**: Grafana dashboard for visualization
- **Structured Logging**: Correlation IDs and detailed logging

## Metrics

### Success Rate Metrics
- `business.modification.attempt` - Total modification attempts
- `business.modification.success` - Successful modifications
- `business.modification.failure` - Failed modifications
- `business.modification.success.rate` - Calculated success rate gauge

### Cost Metrics
- `business.modification.cost` - Modification costs (summary)
- Average cost calculated per modification type

### Wallet Transaction Metrics
- `business.modification.wallet.transaction` - Transaction count by type (DEBIT/CREDIT)
- `business.modification.wallet.transaction.amount` - Transaction amounts (summary)

### API Error Metrics
- `business.modification.api.error` - API errors by error type
- `business.modification.api.error.rate` - Calculated error rate gauge

### Processing Time Metrics
- `business.modification.processing.duration` - Processing time histogram with p50, p95, p99 percentiles

### Rollback Metrics
- `business.modification.rollback` - Rollback count
- `business.modification.rollback.rate` - Calculated rollback rate gauge

## Alert Rules

The `modification-alerts.yml` file contains Prometheus alert rules:

### Critical Alerts (Immediate Action Required)
- **ModificationSuccessRateCritical**: Success rate < 90% for 2 minutes
- **ModificationApiErrorRateCritical**: API error rate > 10% for 2 minutes
- **ModificationRollbackRateCritical**: Rollback rate > 5% for 2 minutes
- **ModificationProcessingTimeCritical**: p95 processing time > 10 seconds for 2 minutes

### Warning Alerts (Investigation Needed)
- **ModificationSuccessRateLow**: Success rate < 95% for 5 minutes
- **ModificationApiErrorRateHigh**: API error rate > 5% for 5 minutes
- **ModificationRollbackRateHigh**: Rollback rate > 1% for 5 minutes
- **ModificationProcessingTimeSlow**: p95 processing time > 5 seconds for 5 minutes

### Informational Alerts
- **InsufficientBalanceFailuresHigh**: High rate of insufficient balance failures
- **ModificationTypeFailureRateHigh**: High failure rate for specific modification type
- **NoModificationActivity**: No modifications for 30 minutes

## Grafana Dashboard

The `modification-dashboard.json` file contains a Grafana dashboard with:

1. **Success Rate Graph**: Real-time success rate with 95% threshold alert
2. **Attempts by Type**: Breakdown of modification attempts by type
3. **API Error Rate**: API error rate with 5% threshold alert
4. **Rollback Rate**: Rollback rate with 1% threshold alert
5. **Processing Time Percentiles**: p50, p95, p99 latency tracking
6. **Average Cost**: Average modification cost by type
7. **Wallet Transaction Volume**: Debit vs credit transaction rates
8. **Failure Reasons**: Pie chart of failure reasons
9. **Success Rate by Type**: Table showing success rate per modification type

## Setup Instructions

### 1. Prometheus Configuration

Add the alert rules to your Prometheus configuration:

```yaml
# prometheus.yml
rule_files:
  - /path/to/modification-alerts.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
```

### 2. Grafana Dashboard Import

1. Open Grafana UI
2. Navigate to Dashboards → Import
3. Upload `modification-dashboard.json`
4. Select your Prometheus data source
5. Click Import

### 3. Application Configuration

The metrics are automatically exposed via Spring Boot Actuator at:
- Metrics endpoint: `http://localhost:8081/actuator/metrics`
- Prometheus endpoint: `http://localhost:8081/actuator/prometheus`

Ensure your `application.yml` includes:

```yaml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  metrics:
    export:
      prometheus:
        enabled: true
```

### 4. Alert Manager Configuration

Configure AlertManager to send notifications:

```yaml
# alertmanager.yml
route:
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'team-notifications'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'team-notifications'
    email_configs:
      - to: 'team@example.com'
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<your-pagerduty-key>'
  - name: 'slack'
    slack_configs:
      - api_url: '<your-slack-webhook>'
        channel: '#alerts'
```

## Structured Logging

All modification operations include structured logging with:
- **Correlation IDs**: Unique identifier for each modification attempt
- **Agent ID**: User performing the modification
- **Agency ID**: Agency context
- **Modification Type**: Type of modification
- **Booking ID**: Booking being modified
- **Timestamps**: Request and completion times
- **Error Details**: Detailed error information for failures

Example log entry:
```json
{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "INFO",
  "logger": "ModificationServiceImpl",
  "message": "Successfully completed modification",
  "bookingId": "123e4567-e89b-12d3-a456-426614174000",
  "modificationId": "mod-789",
  "modificationType": "FLIGHT_DATE",
  "agencyId": "agency-456",
  "agentId": "agent-123",
  "cost": 200.00,
  "transactionType": "DEBIT",
  "duration": 1234
}
```

## Querying Metrics

### Prometheus Queries

**Success rate over last 5 minutes:**
```promql
(sum(rate(business_modification_success_total[5m])) / sum(rate(business_modification_attempt_total[5m]))) * 100
```

**Average processing time by type:**
```promql
avg(rate(business_modification_processing_duration_seconds_sum[5m])) by (type) / avg(rate(business_modification_processing_duration_seconds_count[5m])) by (type)
```

**Wallet transaction volume:**
```promql
sum(rate(business_modification_wallet_transaction_total[5m])) by (type)
```

**API error rate:**
```promql
(sum(rate(business_modification_api_error_total[5m])) / sum(rate(business_modification_attempt_total[5m]))) * 100
```

## Troubleshooting

### High Failure Rate
1. Check `business.modification.failure.by.type` for failure reasons
2. Review application logs for error details
3. Check Amadeus API status
4. Verify wallet balances

### High API Error Rate
1. Check Amadeus API health
2. Review `business.modification.api.error` by error type
3. Check network connectivity
4. Verify API credentials

### High Rollback Rate
1. Review transaction logs
2. Check database connectivity
3. Verify wallet service health
4. Check for concurrent modification issues

### Slow Processing Time
1. Check database query performance
2. Review Amadeus API response times
3. Check system resource utilization
4. Look for lock contention

## Requirements Validation

This monitoring implementation validates:
- **Requirement 8.1**: All modification attempts are logged with agent, timestamp, and type
- **Requirement 8.2**: Success operations record before/after states
- **Requirement 8.4**: Failures are logged with detailed error information

## Maintenance

- Review alert thresholds quarterly based on actual performance
- Update dashboard panels as new metrics are added
- Archive old metrics data according to retention policy
- Test alert notifications regularly