The interaction of mental representations from linguistic and visual input: A processing account of role-assignment in visual worlds

Pia Knöferle$^1$, Matthew W. Crocker$^1$, Christoph Scheepers$^2$ & Martin J. Pickering$^3$
{knoeferle,crocker}@coli.uni-sb.de, C.Scheepers@dundee.ed.ac.uk, M.Pickering@ed.ac.uk
1 Universität des Saarlandes; 2 University of Dundee; 3 University of Edinburgh

While much work has examined referent selection and the effects of referential properties of objects, little is known about the use of other ontological categories (e.g. actions and events) and their role-assigning effects in sentence comprehension. We report four studies using eye-movements in visual scenes to investigate role-assignment and structural disambiguation through actions depicting role-relations between agents and patients in agent-action-patient events in initially structurally ambiguous spoken sentences.

All experiments exploited similar pictures. Images depicted 2 events each involving 2 of 3 characters. One character was role-ambiguous (patient/agent), engaged in 2 events; the other two were unambiguously agent and patient, respectively, e.g.:

CELLIST	$\leftarrow$splashing--	BALLERINA	$\leftarrow$sketching--	FENCER
(Patient)		(Patient/Agent)		(Agent)

Unlike previous studies, visual stimuli offered only depicted role and event information. Selectional/stereotypical knowledge was not available for disambiguation.

Experiments 1 and 2 investigated the comprehension of initially structurally ambiguous spoken German SVO/OVS main clauses (NP1-V-ADV-NP2). While the NP1-V sequences alone did not disambiguate, the depicted actions showed who-did-what-to-whom. In Experiment 1, shortly after the verb and before disambiguation by NP2 case marking, we observed anticipatory inspections to the patient for SVO, and to the agent for OVS sentences. In addition we found effects of initial role-interpretation and role-reinterpretation processes. Experiment 2 reconfirmed the findings while controlling for intonation. Experiment 3 established that depicted actions facilitated initial interpretation of role-relations in verb-final constructions.

Experiment 4 replicated findings for German SVO/OVS clauses for a different construction (MV/RR) and language (English). We contrasted furthermore lexical (main verb/past-part) information (1) with purely functional (aux verb) information (2) in on-line disambiguation. Sentences had the following form:

1

a.

The ballerina splashed quickly the cellist in the white shirt.

b.

The ballerina sketched quickly by the fencer splashes the cellist.

2

a.

The ballerina will quickly splash the cellist.

b.

The ballerina being quickly sketched by the fencer splashes the cellist.

On the assumption that purely functional information may allow a more rapid disambiguation than lexical (verb) information, which only disambiguates together with event depictions, we predicted an earlier disambiguation effect for sentences (2) vs. (1).

Our findings suggest that depicted actions are used for rapid disambiguation of local structural and role ambiguity which verb knowledge alone could not have disambiguated. We provide an account of the interaction between linguistic and visual information in on-line sentence comprehension on the basis of conceptual semantics (cf. Jackendoff, 1983, 1990), bringing together insights from theoretical semantics and experimental psycholinguistics.

References

Jackendoff, Ray. 1983. Semantics and Cognition. Cambridge, Mass.: MIT Press.

Jackendoff, Ray. 1990. Semantic Structures. Cambridge, Mass.: MIT Press.