McGurk effect

Play a recording of “ba” over a face mouthing “ga.” Some listeners hear “da,” a syllable supplied by neither track alone. Others hear “ba.” The lips can change the heard sound, but the illusion is not inevitable. What rule makes a listener combine the conflicting clues or keep them apart?[1] [2]

A face mouthing “ga” can change a recording of “ba” into a heard “da,” yet some listeners keep hearing “ba.” The question is which rule converts conflicting sound and lip movements into a syllable report, and whether distinct neural estimates implement that rule.

The listener has two candidate accounts of the clip: one talker whose sound and mouth movements belong together, or separate sources whose cues should be kept apart. Judging which account generated the clues is causal inference. Write the accounts as C = 1 and C = 2.

In the 2017 CIMS model, C = 1 produces a reliability-weighted audiovisual estimate; C = 2 uses the auditory estimate. Model averaging blends these estimates using the probabilities of the two accounts, then assigns a syllable category. Alternatives select the more likely account or sample accounts in proportion to their probabilities. These rules combine internal estimates, rather than mixing physical sound tracks.[1] [2]

When the lips and sound disagree, does the listener blend possible interpretations, choose the most likely one, or switch between them across trials? The rule remains unsettled: several rules can produce similar syllable reports in a small test.[1] [2]

Compare these strategies on a wider set of syllables, independently varying the sound–mouth mismatch and their timing. Constrain sensory noise with sound-only and vision-only trials, and predict each listener's reports on clips excluded from model fitting. The target is a rule that predicts the pattern of reports, not merely the overall number of “da” responses.

Also test whether distinct neural representations correspond to the common-source and separate-source estimates. A 2026 EEG study found candidate processing stages in Mandarin-speaking boys, with and without autism; it did not compare the three inference rules or establish those representations causally.[1] [2]

A face is normally useful evidence about what someone is saying, especially in noise. This illusion gives researchers a controlled way to ask when that evidence helps, when it changes a sound, and why two people can hear the same clip differently.[1] [2]

A model can reproduce syllable choices while giving the wrong account of how they were produced. Linking its separate estimates and combination rule to neural activity would turn a useful behavioral fit into a testable processing explanation.

An intervention would be most informative if it changed the predicted combination rule while leaving basic sound and lip-movement recognition intact. That would help separate a change in cue encoding from a change in how cues are combined. Evidence from one syllable task or child sample still needs separate tests before extending it to everyday conversation.[1] [2]