# Pipeline

## Purpose

This document maps the main analysis pipelines from source datasets to
intermediate outputs and final artifacts. It is an operational reference for
reproducing key deliverables and understanding which script produces which file.

For the registry of concrete tracked outputs, see
[findings/data/index.csv](findings/data/index.csv:1)
and [docs/data-inventory.md](docs/data-inventory.md:1).

## Core Rule

Each pipeline should be traceable in this form:

`input dataset -> script -> output file -> downstream artifact`

If an output exists in `findings/data/` or `artifacts/` but no corresponding
pipeline is documented here, treat it as documentation debt.

## Pipeline Map

### Vault credentials

Input:
- `vault/*/passwords.json`

Script:
- [scripts/vault_creds_export.py](scripts/vault_creds_export.py:1)

Primary output:
- `findings/data/vault_creds.csv`

Downstream documents:
- [docs/processing/vault_creds_analysis.md](docs/processing/vault_creds_analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)
- [artifacts/executive-summary.md](artifacts/executive-summary.md:1)

### Dumps credential aggregation

Input:
- `dumps/*/credentials.csv`

Script:
- [scripts/aggregate_dumps_creds.py](scripts/aggregate_dumps_creds.py:1)

Primary output:
- `findings/data/dumps_creds_aggregated.csv`

Downstream documents:
- [docs/processing/dumps_creds_analysis.md](docs/processing/dumps_creds_analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### KeePass inventory and cracking queue

Inputs:
- `findings/dumps/*/keepass/*.kdbx`
- `findings/keepass/*.kdbx`
- `findings/data/legacy_keepass_analysis.json`
- `findings/data/kdbx_hashes.txt` (historical current-hash labels, mismatch-checked)

Script:
- [scripts/kdbx_inventory.py](scripts/kdbx_inventory.py:1)

Primary outputs:
- `findings/data/kdbx_inventory.csv`
- `findings/data/kdbx_inventory_summary.json`
- `findings/data/kdbx_hash_buckets/*`

Downstream documents:
- [docs/processing/kdbx_inventory_and_cracking.md](docs/processing/kdbx_inventory_and_cracking.md:1)
- [docs/processing/kdbx_cracking.md](docs/processing/kdbx_cracking.md:1)
- [docs/processing/multi_kdbx_analysis.md](docs/processing/multi_kdbx_analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### GitHub / token validation

Input:
- `findings/data/dumps_creds_aggregated.csv`
- extracted token rows from `dumps/*/credentials.csv`

Scripts:
- [scripts/validate_remaining_tokens.py](scripts/validate_remaining_tokens.py:1)

(Per-provider OAuth/PAT L1+L2 validators such as GitHub token checkers are
written per-engagement with the actual tokens as input; they are not part of
the generic template.)

Primary outputs:
- `findings/data/remaining_tokens_validation.md`

Downstream documents:
- [docs/processing/dumps_creds_analysis.md](docs/processing/dumps_creds_analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### Cookie session analysis

Input:
- `vault/*/cookies.json`

Script:
- [scripts/analyze_cookies.py](scripts/analyze_cookies.py:1)

Primary outputs:
- `findings/data/cookie_analysis.tsv`
- `findings/data/cookie_analysis.md`

Downstream documents:
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)
- [artifacts/gap-analysis.md](artifacts/gap-analysis.md:1)

### Credit card analysis

Input:
- `vault/*/credit_cards.json`

Script:
- [scripts/analyze_credit_cards.py](scripts/analyze_credit_cards.py:1)

Primary output:
- `findings/data/credit_card_analysis.md`

Downstream documents:
- [artifacts/digital-asset-impact.md](artifacts/digital-asset-impact.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)
- [artifacts/gap-analysis.md](artifacts/gap-analysis.md:1)

### Monetary impact

Inputs:
- `findings/data/vault_creds.csv`
- `findings/data/credit_card_analysis.md`
- `docs/processing/*`
- selected `findings/data/*_remote_access.md`

Script:
- [scripts/monetary_impact.py](scripts/monetary_impact.py:1)

Primary output:
- `findings/data/monetary_impact.md`

Downstream documents:
- [artifacts/digital-asset-impact.md](artifacts/digital-asset-impact.md:1)
- [artifacts/executive-summary.md](artifacts/executive-summary.md:1)
- [artifacts/gap-analysis.md](artifacts/gap-analysis.md:1)

### Batch environment parsing

Input:
- `batches/*`

Script:
- [scripts/parse_env_batches.py](scripts/parse_env_batches.py:1)

Primary output:
- `findings/data/batch_env_vars.csv`

Downstream documents:
- [docs/processing/batches_analysis.md](docs/processing/batches_analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### Fresh breach extraction

Inputs:
- `findings/telegram_logs/boxed_pw/*`
- extracted `All Passwords.txt` trees

Scripts:
- [scripts/intake_local.py](scripts/intake_local.py) — local archive → extract → aggregate → corp_search → index (replaces the legacy `boxed_pw_pipeline.py` which was folded into `intake_local.py`)
- [scripts/aggregate_breach.py](scripts/aggregate_breach.py)

Primary outputs:
- `findings/telegram_logs/boxed_pw/*_creds.tsv`
- `findings/data/DAISYCLOUD-5950.tsv`
- `findings/data/PIXELCLOUD3-7037.tsv`
- `findings/data/VALENCIGA-534.tsv`
- `findings/data/EVOLUTION-66-1100.tsv`

Downstream documents:
- `findings/data/*_remote_access.md` (manual analysis; see pipeline gap below)
- `findings/data/fresh_breach_verification.md` (not yet created — see Gaps)
- `findings/data/victim_pivot_results.md` (not yet created — see Gaps)

### Corporate search

Inputs:
- cloud log dumps
- extracted breach TSVs

Script:
- [scripts/corp_search.py](scripts/corp_search.py:1)

Primary output:
- `findings/data/corp_search_results.md`

Downstream documents:
- `findings/data/fresh_breach_verification.md` (not yet created — see Gaps)
- [artifacts/executive-summary.md](artifacts/executive-summary.md)

### Telegram session analysis

Inputs:
- `dumps/*/tdata`

Scripts:
- [scripts/extract_tg_hashes.py](scripts/extract_tg_hashes.py)
- [scripts/tg_session_load.py](scripts/tg_session_load.py) — replaces the legacy `validate_tg_sessions.py` / `tg_session_checker.py` names referenced in earlier drafts; those scripts were never created under those names

Primary outputs:
- `findings/data/tg_hashes.txt`
- `findings/data/tg_session_validation.json`

Downstream documents:
- [artifacts/gap-analysis.md](artifacts/gap-analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### WingsCloud ULP intake

Inputs:
- channel «Wings Daily Updates FREE» (invite-only; MTProto session via
  [scripts/tg_session_load.py](scripts/tg_session_load.py:1)).
  Channel profile: [docs/processing/wingscloud_osint.md](docs/processing/wingscloud_osint.md:1)

Rules:
- only files with `ULP` marker in the name (skip combo/user:pass batches)

Scripts:
- [scripts/tg_download.py](scripts/tg_download.py:1) — resumable download (offset-resume, DC-migration)
- [scripts/ulp_to_tsv.py](scripts/ulp_to_tsv.py:1)

Primary outputs:
- `findings/telegram_logs/wingscloud/*.txt` (raw)
- `findings/data/WINGSCLOUD-ULP-<BATCH>.tsv`
- `findings/data/corp_search_WINGSCLOUD-ULP-<BATCH>_results.md`
- `findings/data/WINGSCLOUD-ULP-<BATCH>_rce_candidates.tsv`
- `findings/data/WINGSCLOUD-ULP-<BATCH>_api_keys.tsv`

Downstream documents:
- `findings/data/WINGSCLOUD-ULP-<BATCH>_priority_matrix.md`
- [artifacts/executive-summary.md](artifacts/executive-summary.md:1)

### DIY bot collection (P0 pilot)

Context: [docs/log-source-discovery.md](docs/log-source-discovery.md:1) §5.2, §8

Scripts:
- [scripts/tg_bot_harvest.py](scripts/tg_bot_harvest.py:1) — ThreatFox/URLhaus/MB → bot tokens → getMe validation
- [scripts/tg_bot_collect.py](scripts/tg_bot_collect.py:1) — getUpdates daemon → per-victim files
- [scripts/tg_group_read.py](scripts/tg_group_read.py:1) — exfil-group history via exportChatInviteLink + MTProto

Primary outputs:
- `findings/data/tg_bots_validated.tsv` (tokens, full values — no-masking)
- `findings/data/tg_bots_state.json` (collector offsets, discovered_chats)
- `findings/breaches/TGBOT-<bot_id>/` (messages, raw JSON, documents)
- `findings/data/tg_group_joins.log` (OPSEC audit of group joins)

Downstream: стандартный `intake_local.py --extracted findings/breaches/TGBOT-<id>`
→ corp_search/probe_rce/scan_secrets.

### Password reset chain inference
Input:
- `findings/data/vault_creds.csv`

Script:
- [scripts/password_reset_chain.py](scripts/password_reset_chain.py:1)

Primary outputs:
- `findings/data/password_reset_chain_graph.csv`
- `findings/data/password_reset_chain_summary.md`

Downstream documents:
- [artifacts/gap-analysis.md](artifacts/gap-analysis.md:1)
- [artifacts/credential-scope-assessment.md](artifacts/credential-scope-assessment.md:1)

### Status synchronization

Input:
- `artifacts/status-matrix.csv`

Scripts:
- [scripts/generate_status_summary.py](scripts/generate_status_summary.py:1)
- [scripts/validate_status_matrix.py](scripts/validate_status_matrix.py:1)
- [scripts/status_report.py](scripts/status_report.py:1)

Primary effects:
- managed block update in [artifacts/gap-analysis.md](artifacts/gap-analysis.md:278)
- validation of narrative/CSV consistency
- read-only operational summary

Operational entrypoints:
- `make sync-status`
- `make status-report`

## Gaps

Pipelines still insufficiently documented:
- Firefox `key4.db + logins.json` decrypt flow
- `vault/autofill.json` -> PII quantification flow
- exact path from fresh-breach TSVs to each `*_remote_access.md`
- repeatable path for `threat-scenarios.md` synthesis

Pipeline debt (resolved + open):
- `boxed_pw_pipeline.py` was referenced here but never existed as a
  standalone script; its function was folded into `intake_local.py`.
  Reference corrected above (2026-07-13).
- `validate_tg_sessions.py` and `tg_session_checker.py` were referenced
  here but never created under those names; the active script is
  `tg_session_load.py`. Reference corrected above (2026-07-13).
- `findings/data/fresh_breach_verification.md` and
  `findings/data/victim_pivot_results.md` were listed as downstream
  outputs but have not been created yet. Their content is currently
  spread across `findings/data/*_L1L2_validation.md` and the per-target
  dossiers under `redteam/`. Consolidating these into the canonical
  files is open documentation debt.

These should be added here once their processing logic is stabilized.