Word and image

In this unit you will look at ways in which visual aspects add further layers of meaning to printed literary texts and may be considered to contribute to their poetic function.

As you read, look out for:

• how multimodal elements can be seen to contribute to the literariness of texts by foregrounding certain aspects through deviation and defamiliarisation;
• the application of the theory of semiotics as conceived by Saussure and developed by Peirce (see Reading A by Bignell), and how the interpretation of signs and codes depends upon the social and cultural knowledge of individual readers;
• the use of visual images with verbal texts in the creation of fictional reality in both children’s and adult narratives. (Note that the term ‘high modality’ in this context refers to the feeling of truth and reality conveyed and should not be confused with ‘modes’ of communication or ‘multimodality’);
• the way words and images are used in some children’s picturebooks to
  create ‘hybrid’ texts that challenge and subvert established ideological notions of history. In Reading B, Bradford analyses two children’s texts to show how they ideologically disrupt dominant postcolonial perceptions;
• how children’s picturebooks may combine words and images in distinctly postmodern ways to create strange and uncertain meanings. See Lewis’ discussion of the features of postmodernity in Reading C, and the analysis of examples in his reading and in the main chapter;
• arguments about how and why we make aesthetic judgements about multimodal texts especially in relation to their material form and context.

The use of different forms of communication in a single text is often known as multimodality. I am primarily concerned here with texts utilising image and words, from the genres of poetry and narrative. Texts from other genres, such as advertisements, and texts using other modes of communication, such as movement, are also included where they seem particularly salient and can be seen as literary, in the Formalist sense. It is important to look both at how individual modes can communicate meaning, and at their interaction in the text – how they can be combined to open up new possibilities for interpretation. Such combinations and juxtapositions of modes can be seen to enhance, reinforce or contradict each other, making meaning unstable and challenging the reader’s attempts to make sense of the text.

Multimodality is researched in a range of academic disciplines, among them linguistics, art history, information technology, sociology, cultural studies and media studies. Multimodal texts tell us something about linguistic and artistic creativity, but also about the social and cultural spheres in which we live, and about ourselves as producers and consumers of texts. Close analysis opens up questions about how we make judgements about visual texts, about how we value or downplay their status. Creative multimodality also tells us something about how language works – it may force us to focus on a metaphor or a pun, for example. Or it may reveal something about how relationships between people and institutions are represented, or just cause us to reflect on something we had forgotten to notice for a while.

New technology has brought about a surge in multimodal creativity, as anyone with a reasonably up-to-date computer can now produce texts filled with images, sound and movement. But texts using visuals to make meaning are far from new.

Activity 1: The Mouse’s Tale

You should allow 0 hour(s), 10 minute(s).

Start by considering the well-known example of ‘The Mouse’s Tale’ from Lewis Carroll’s Alice’s Adventures in Wonderland (1865), shown in Figure 1. First, what is this? A poem, a narrative, a picture?

Figure 1: ‘The Mouse’s Tale’ (Lewis Carroll, 1865, p. 38).

Think about how the physical shape of the printed text influences your interpretation. It might be helpful to imagine it laid out differently. If the text were set out in the usual, left-aligned layout of prose or some other poems, would you ‘read’ it differently? Why?

Now read the answer

Comment

‘The Mouse’s Tale’ is an example of concrete poetry, where words are arranged on the page in a significant way. When read aloud, its rhyme and metre are easier to access, but the layout initially challenges our attempts to read it as a poem. We consider concrete poetry in more detail in the next section.

‘The Mouse’s Tale’ can be considered as literary not only because it comes from a highly valued and canonical work of literature, but because it can be analysed in terms of defamiliarisation and creative deviation in the discussion of Russian Formalism. The poem is eye-catching (‘made strange’) due to its unusual shape and can thus be seen as deviating from ‘normal’ layout conventions of poetry – even though readers of poetry are familiar, of course, with a range of such conventions. The text is also poetic in Formalist terms because it contains an obvious pun – in this case between semiotic modes. The verbal ‘tale’ being told by the mouse is represented by the visual ‘tail’.

Multimodality forces us as readers to focus on ‘the message for its own sake’ (Jakobson, 1960, p. 356) and pay fresh attention to departures from routinised conventions of literature or poetry. Victor Shklovsky believed that the purpose of art was exactly this – to counter our habitualised perceptions, to force us to notice: