# CP-8 Audio + Lip-Sync Implementation Audit

**Generated:** 2026-04-28 08:42 EDT
**Anchor tag:** `pre-cp8-audio-lipsync`
**Stage:** Phase 1 (audit only; ships zero code).
**Caller catalog:** `consultations/recoil/cp8-audio-lipsync-spec/cp8_phase1_audio_callers.json`

## 1. Existing audio surface

### 1.1 CP-4 stub runners (the seats CP-8 fills)
- `pipeline/core/runners/audio_runner.py:27-33` — `AudioRunner` stub, `modality = MODALITY_AUDIO_T2A`, `run()` raises `NotImplementedError("Audio modality (audio_t2a) lands in CP-8.")`.
- `pipeline/core/runners/lipsync_post.py:18-24` — `LipSyncPostProcessor` stub, `modality = MODALITY_LIPSYNC_POST`, `run()` raises `NotImplementedError("Lip-sync post-processor (lipsync_post) lands in CP-8.")`.
- Both stubs already reference sync.so / ElevenLabs in their docstrings: `audio_runner.py:13`, `lipsync_post.py:6`.

### 1.2 Modality constants (locked names)
- `pipeline/core/registry.py:47` — `MODALITY_AUDIO_T2A = "audio_t2a"`.
- `pipeline/core/registry.py:48` — `MODALITY_LIPSYNC_POST = "lipsync_post"`.
- Re-exported via `pipeline/core/__init__.py:31-32` and `pipeline/core/runners/__init__.py:18-19`.

### 1.3 Registration sites (idempotent stub registration — frozen contract)
- `pipeline/core/runners/__init__.py:35-38` — `_register_stubs_once()` registers both stubs at package import (zero-arg constructors).
- `pipeline/core/dispatch.py:85-100` — `register_default_runners(force=...)` re-registers `AudioRunner()` + `LipSyncPostProcessor()` idempotently.
- Phase 4 will swap stubs for real runners at THESE SAME registration sites — names, modality keys, and zero-arg constructors stay byte-identical.

### 1.4 Test surface (must continue to pass green throughout CP-8)
- `pipeline/core/tests/test_registry.py:46-47` — modality constant equality.
- `pipeline/core/tests/test_module_load_registration.py:36-39, 100, 104, 111` — auto-registration on package import; both runners resolvable via `get_runner()`.
- `pipeline/core/tests/test_stub_runners.py:13-71` — full stub contract (modality, NIE on `run()`, register/resolve roundtrip, list_modalities membership). Phase 4 will rewrite this file when stubs become real.
- `pipeline/core/tests/test_dispatch.py:86` — dispatches `audio_t2a` end-to-end.
- `pipeline/core/tests/test_workflow_dispatch_integration.py:223` — workflow integrates `audio_t2a` step.
- `pipeline/core/tests/test_workflow_e2e_scenarios.py:146` — workflow integrates `lipsync_post` step.
- `pipeline/core/tests/test_receipts.py:118-120` — receipt id pattern for `audio_t2a` (suffix `_audio_t2a`).

### 1.5 Workflow integration anchor
- `pipeline/core/workflow.py:46` — comment block notes "image_t2i / video_i2v live; audio_t2a / lipsync_post stub" — Phase 5 updates this comment to reflect "live" status; otherwise no workflow changes required.

### 1.6 Legacy / out-of-scope references (DO NOT TOUCH IN CP-8)
- `visual/compiler.py:405` — task-manifest writes `'model': 'elevenlabs'` (legacy task-manifest path; CP-8 does NOT route through compiler.py).
- `tools/cost_tracker.py:848` — provider category list contains `"elevenlabs"`. Cost tracker is out of scope unless trivial; CP-8 cost_compute is owned by adapters and lands in receipts.
- `tools/fountain_reader.py` lines 24, 28, 33, 57-58, 84, 319, 322, 431, 439, 443, 568, 581, 1177 — long-standing `elevenlabs` Python SDK integration in the screenplay/audiobook reader. Separate pipeline from runners; NOT touched in CP-8. Flagged for future consolidation.
- `tools/engine_checks/documentation.py:184` — checks `latest.get("elevenlabs", {})`; engine-checks layer, untouched.
- `pipeline/config/starsend_config.json:38` — `"tts_provider": "elevenlabs"` (legacy substrate label; not read by runners).
- `config/pipeline_config.json:52` — `"tts_provider": "elevenlabs"` (engine-level config; CP-8 reads model_profiles instead).
- `config/pricing_rates.json:42` — pricing rates section for `elevenlabs` (cost-tracker substrate; out of scope).
- `config/model_roles.json:27` — `"elevenlabs": "eleven_v3"` (NOTE: legacy role mapping says `eleven_v3`; CP-8 locks `eleven_multilingual_v2`. This file is NOT consumed by the runner path so the divergence is harmless within CP-8 scope; flag for future cleanup).

### 1.7 Negative findings (the holes CP-8 fills)
- ZERO occurrences of `ELEVENLABS_API_KEY` env var anywhere in the tree — CP-8 introduces the canonical name.
- ZERO occurrences of `XI_API_KEY` — CP-8 reads `ELEVENLABS_API_KEY` (HTTP header is still `xi-api-key:`).
- ZERO occurrences of `SYNC_SO_API_KEY` — CP-8 introduces the canonical name.
- The only `sync_so` mentions in the entire tree are the two stub runner docstrings (`audio_runner.py:13`, `lipsync_post.py:6`); no client code, no config, no cost_tracker entry.
- `eleven_multilingual_v2` and `lipsync-2.0` are ABSENT from `recoil/config/model_profiles.json` and `recoil/config/provider_strategy.json` — Phase 3 adds them.

## 2. Vendor API surface (full detail in `cp8_phase1_vendor_endpoints.md`)
- ElevenLabs TTS: POST `https://api.elevenlabs.io/v1/text-to-speech/{voice_id}?output_format=mp3_44100_128`, `xi-api-key` header, JSON body `{text, model_id, voice_settings}`. Response: audio bytes.
- sync.so lipsync v2: POST `/v2/upload` (multipart) → URL; POST `/v2/generate` `{model, input[video,audio], options}` → job id; GET `/v2/generate/{id}` poll; download `outputUrl`.
- Error mapping (BOTH): 401/402/422/429 fail-fast; 5xx + network retry 3x exp backoff (1s, 2s, 4s).

## 3. Locked decisions (flagged for spec-review)
1. TTS: ElevenLabs primary, model `eleven_multilingual_v2`.
2. Lipsync: sync.so primary, model `lipsync-2.0`.
3. No StepRunner additions. Runners → providers/ adapters directly.
4. Payload schemas: audio_t2a needs `{shot_id, text, voice_id, model}`; lipsync_post needs `{shot_id, video_path, audio_path, model}`.
5. Output dirs `$RECOIL_ROOT/_audio_outputs/`, `_lipsync_outputs/`.
6. Test boundary: mock at `urlopen` via `transport=` injection.
7. Carrier-video helper deferred.
8. sync.so URL upload via `/v2/upload` (not base64 inline).
9. Env var canonical names: `ELEVENLABS_API_KEY` and `SYNC_SO_API_KEY` (neither currently exists in the tree — clean introduction).
10. Frozen contracts (byte-unchanged through all of CP-8 except where explicitly listed): `pipeline/core/dispatch.py`, `pipeline/core/receipts.py`, `pipeline/core/dispatch_context.py`, `pipeline/core/registry.py`, `pipeline/core/workflow.py`, `pipeline/core/take.py`, `execution/step_runner.py`, `execution/step_types.py`. Phase 4 rewrites the bodies of `runners/audio_runner.py` and `runners/lipsync_post.py`; Phase 1-3 leave them byte-unchanged.

## 4. Risks
- Vendor model versioning drift (verify at dispatch time).
- Cost compute formulas differ (per-1k-chars TTS vs per-second lipsync).
- API keys not yet in JT's .env — Phase 8 manual matrix gates live calls.
- `visual/compiler.py:405` and `tools/fountain_reader.py` legacy ElevenLabs references — flagged for a future CP. CP-8 does not consolidate.
- `config/model_roles.json:27` says `eleven_v3`, deliberate divergence from CP-8's `eleven_multilingual_v2`. This file is NOT read by the runner path so the divergence is harmless within CP-8 scope; flag for cleanup.
- `tools/cost_tracker.py:848` lists `"elevenlabs"` as a provider; CP-8 adapter cost_compute writes through receipts, NOT cost_tracker. Reconcile in a future CP if double-accounting appears.

## 5. Phase 2 hand-off
Phase 2 reads `cp8_phase1_vendor_endpoints.md` for the exact API surface and builds `pipeline/core/providers/elevenlabs.py` + `pipeline/core/providers/sync_so.py` (transport-injectable, urlopen-mockable). Phase 2 does NOT modify the stub runners — Phase 4 swaps the bodies. Phase 3 adds model_profiles + provider_strategy entries between Phase 2 and Phase 4. The frozen contracts list in §3 item 10 is the byte-unchanged guarantee Phases 2-3 must respect.

## Post-CP-8 (Phase 7)

### Shipped surface

- `recoil/execution/providers/elevenlabs.py` — ~280 lines. `synthesize_speech` + SynthesisResult + 6-class exception tree.
- `recoil/execution/providers/sync_so.py` — ~340 lines. `lipsync_video` + LipSyncResult + 8-class exception tree (incl. JobFailedError, JobTimeoutError).
- `recoil/pipeline/core/runners/audio_runner.py` — ~150 lines (rewrite-in-place from 37-line stub).
- `recoil/pipeline/core/runners/lipsync_post.py` — ~160 lines (rewrite-in-place from 29-line stub).
- `recoil/config/model_profiles.json` — +2 entries: `eleven_multilingual_v2`, `lipsync-2.0`.
- `recoil/config/provider_strategy.json` — +2 entries.
- 7 new test files, ~85 test cases total.
- `recoil/_audio_outputs/` + `recoil/_lipsync_outputs/` directories (gitignored).

### Deferred items (intentional scope cuts)

1. Carrier-video helper (Seedance black-screen trick from SYNTHESIS) — caller supplies video_path. Phase 1 audit documents the gap.
2. WorkflowStep payload templating (`$tts.output_path`) — pre_step hook is the documented workaround. Future CP can add native templating.
3. Streaming TTS — only batch synthesis in CP-8.
4. Multi-character lipsync — single video + single audio per call.
5. compiler.py legacy task-manifest migration — out of scope; flagged for future CP.
6. cost_tracker.py integration — RunResult.metadata.cost_usd is authoritative; cost_tracker integration is incremental.
7. ProviderAdapter Protocol unification — adapter modules are functional, not registered in `execution/providers/registry.py`. Future CP can promote.
8. StepRunner.execute_audio / execute_lipsync — explicitly deferred per JT default.

### Frozen contracts (NOT modified)

CP-3 prompt engine, CP-4 RunResult/registry, CP-5 GenerationReceipt/dispatch/DispatchContext, CP-6 Workflow/WorkflowStep, CP-7 Take/Beat/Scene, StepRunner / step_types — all byte-stable.

### Rollback

```bash
git checkout pre-cp8-audio-lipsync -- \
    recoil/pipeline/core/runners/audio_runner.py \
    recoil/pipeline/core/runners/lipsync_post.py \
    recoil/config/model_profiles.json \
    recoil/config/provider_strategy.json \
    recoil/.gitignore
git rm recoil/execution/providers/elevenlabs.py recoil/execution/providers/sync_so.py
git rm recoil/pipeline/core/tests/test_audio_runner.py \
       recoil/pipeline/core/tests/test_lipsync_post_runner.py \
       recoil/pipeline/core/tests/test_audio_provider_elevenlabs.py \
       recoil/pipeline/core/tests/test_lipsync_provider_sync_so.py \
       recoil/pipeline/core/tests/test_audio_lipsync_model_profiles.py \
       recoil/pipeline/core/tests/test_audio_lipsync_dispatch.py \
       recoil/pipeline/core/tests/test_audio_lipsync_workflow.py \
       recoil/pipeline/core/tests/test_audio_lipsync_take_beat.py \
       recoil/pipeline/core/tests/test_audio_lipsync_e2e_scenarios.py
git checkout pre-cp8-audio-lipsync -- recoil/pipeline/core/tests/test_stub_runners.py
```