Audio alone can't tell you which face a voice belongs to. Video alone can't tell you whether a moving mouth is producing the sound you hear. This page runs a real audio-visual correlator on your webcam and microphone: per-face lip motion, live audio energy, and the correlation between them — every signal it uses is plotted below as it is computed. Best with two people in frame.
The live demo needs your camera and microphone. Everything runs locally in your browser — no video or audio ever leaves this page.
Per face, over a ~1.6 s sliding window:
score = max(0, corr(lip, audio)) × VAD
Pearson correlation of z-scored lip motion against z-scored audio energy, gated by voice activity
(adaptive noise floor).
This demo does not run the PyTorch model from the repo. No trained weights are shipped here. What runs in your browser is the honest lightweight version of the same idea: MediaPipe face landmarks give a per-face mouth-openness signal, the Web Audio API gives the microphone's energy envelope, and the speaking score is the correlation between the two over a sliding window, gated by voice activity.
That is the core intuition of audio-visual fusion — a face is "speaking" exactly when its mouth motion co-varies with the audio — demonstrated with signals you can inspect, instead of a canned animation. The research model below learns this alignment from data; the demo computes it geometrically, which is why it fits in one HTML file and runs at 15 fps on a laptop CPU.
The model in model.py
processes each modality in its own convolutional tower and fuses by concatenation. Audio: a 3-second window around
the target frame, as 13-coefficient MFCCs — effectively a spectrogram image. Visual: a 112×112 grayscale face crop.
Each tower is four Conv2d blocks with BatchNorm and max-pooling; the flattened embeddings
(4736 audio + 25088 visual = 29824) feed a fully connected head trained with cross-entropy.
createFusionModel is left as a placeholder for a
learned fusion module. The 2024 research focus was the visual tower: replacing it
with VGG16 via transfer learning and freezing selected layers to control fine-tuning, maximizing mAP on AVA
Active Speaker format data. Measured results live in the
research presentation.A narrator, a voice from another room, a phone on speaker: rich speech audio with no correct face to pick. Audio-only systems must hallucinate an answer; an audio-visual model can output "nobody visible is speaking." Try it: play music at the demo with your mouth still.
With five people in frame, the waveform says someone is talking — never who. The decision is inherently per-face: each candidate's lip motion must be tested against the same audio, which is exactly what the per-face scores above do.
Chewing, smiling, yawning, silent mouthing, dubbed or lip-synced video: vision-only systems fire on all of them. The correlation gate fails these cases correctly — motion without matching audio scores near zero.