VOICE AI & FACIAL ENGINE.
The real-time voice agent and facial animation engine I built as one component of Lumina, a larger client platform. Users just talk, in any of 97 languages, and a 3D avatar answers with live lip-sync, emotion and gaze driven by the AI's streaming speech.
THE
ARCHITECTURE
Every design choice starts from latency. Audio streams directly between the browser and the Gemini Live API with no server relay — relaying would have added roughly 280ms to every turn. The FastAPI backend only mints short-lived, single-use ephemeral tokens with the session configuration locked server-side, so the browser can talk to the model but cannot change who the model is. Nothing on the voice path is ever awaited, and a cold start can only ever land on the token request, never mid-conversation.
Around that hot path the backend does the operational work: versioned prompts with an admin console (edit, history, restore), hashed API keys with origin allowlists and per-minute and monthly quotas, a transcript and metrics ingest that answers 202 immediately and is written in batches by a background writer, and per-turn cost tracking priced from the provider's real usage data. A WebSocket side channel streams the rolling transcript to a lightweight "director" model that returns affect and gaze cues for the face.
Phoneme-anchored lip-sync. Text supplies which sound is spoken; audio supplies when. Server-side grapheme-to-phoneme produces IPA phones, and a duration-explicit semi-Markov Viterbi forced aligner places them on the streaming audio. If alignment fails the engine degrades gracefully — acoustic-only, then neutral — instead of freezing the mouth.
A mixer, not layers fighting each other. No layer writes final values. Lip-sync, emotion, gaze, blink and idle each contribute to 167 facial channels, and per-channel policies (additive-clamped, reflex-max, speech-dominant) resolve conflicts before constraints and spring smoothing — so a smile never breaks a lip closure.
Avatar independence. The engine contains no three.js and no avatar-specific code. It talks to an AvatarAdapter contract, runs headless in Node, and every tunable lives in calibration data — a new character needs an adapter and a calibration file, not an engine change.
Real-time discipline. The engine runs at 60Hz with zero per-frame allocation (preallocated typed arrays, dirty masks). Timing is sampled from the audio clock, not accumulated frame time, so the mouth never drifts from the voice. The debug UI reads a shared bus at 20Hz, keeping React off the hot path entirely.
Key
Technical Decisions
Engineering
Range
01. Real-Time Audio & DSP
16kHz PCM capture, gapless 24kHz playback scheduling, FFT feature extraction in a Web Worker ahead of playback, and VAD turn-taking tuned against latency budgets.
02. Speech & Linguistics
IPA phonemization, a viseme taxonomy, coarticulation modelling with per-viseme resistance, and a forced aligner that works on streaming speech in any language.
03. 3D Graphics & Animation
three.js morph targets and bone routing, plus GLB tooling that inventories a rig, measures mouth and eye geometry in millimetres, and shrinks the model from 142 MiB to ~17 MB.
04. AI Integration
Gemini Live API sessions with ephemeral tokens, a secondary director model for affect and gaze, context compression for long conversations, and latency-first configuration.
05. Backend & Data
FastAPI and SQLModel services: prompt versioning with an admin UI, API keys and quotas, and a non-blocking transcript and metrics pipeline on PostgreSQL.
06. Cloud & Cost
Cloud Run and a Firebase Hosting CDN with secrets in Secret Manager, and per-turn cost reconciliation against the provider's billing data.
INTERFACE GALLERY
Diagnosing Lip-Sync From Rendered Frames
Mouth aperture measured from rendered video, turn by turn.
Modulation Spectrum vs. Syllabic Band
Movement peaked near 1Hz — far below the 4–6Hz rhythm of speech.
TECHNICAL
CHALLENGES
01. The Mouth Wasn't Speaking
An early lip-sync looked busy but wrong. Measuring mouth aperture from rendered frames showed movement peaking around 1Hz while syllables arrive at 4–6Hz. That finding drove the rebuild around phoneme anchoring and coarticulation.
02. Interruptions Mid-Word
When a user cuts in, audio playback, the viseme timeline and the five-state conversation machine all reset together, so the avatar stops speaking mid-word and visibly starts listening instead of mouthing words that were never played.
03. Animation State Bugs
The three.js animation mixer snapped the rig back to a T-pose between clips — fixed by posing it at the clip's first keyframe before playback. A state-machine fall-through made the avatar look away mid-sentence; gaze is now driven explicitly per conversation state.
IMPACT & METRICS