Exploring multimodal texts

In a nutshell

The way language is used is evolving in new and interesting ways as a result of new technologies. Media text types are often multimodal, which you will notice when examining them. Multimodal texts can be found in a variety of formats, such as hard copies or digital/electronic copies. Two or more modes are typically integrated to produce meaning in these types of texts. In this summary, different modes and the features of multimodal texts will be explored in more detail.

Semiotic systems

Semiotics is the study of the process of producing meaning from signs and symbols. In multimodal texts, the semiotic system can be used to study and understand the different modes of communication. The semiotic system shows how non-linguistic features of a text create meaning.

Spatial

The arrangement of objects in a space is referred to as the spatial mode. It also addresses the physical proximity of things and people.

Orthographical (linguistic)

The orthographical mode encompasses spoken and written language and comprises aspects such as vocabulary, grammar structure etc. It is the most prevalent form because of its versatility.

Visual

Marks, signs, and symbols are examples of visual elements that fall under this category. The use of colour, layout, style, size and perspective are all part of the visual mode. Consider (still or moving) images on a website; they don't employ any language or sound, thus they seem to be entirely visual. Signs rely on visual communication to convey their message.

Gestural

The gestural mode is concerned with motion and how we perceive it. Facial expressions, hand gestures, body language and interpersonal interactions are all examples of gestural modes. When speaking face-to-face, watching videos on YouTube and listening to live performances, gesturing is important.

Audio

The aural or audio mode emphasises sound. This comprises background noises, music, sound effects, ambient noises, a person's tone of voice, volume, word emphasis and even accent.

When two or more of these systems are combined, new or enhanced meaning is produced. Since many modes must be employed to create or enhance meaning, the majority of the texts we use—including picture books, textbooks, comic books, movies, e-posters, web pages and online videos—are multimodal.

For instance, in order to convey a message and produce an effect, comic books blend spatial, orthographic and visual systems.

Types of multimodal texts

There are four types of multimodal texts:

Type	Explanation	Examples
Digital	A virtual text that combines different modes, such as those found online as well as offline.	Films, ebooks, blogs, vlogs, video games etc.
Live	A live performance of a text.	Theatre, poetry readings, lectures etc.
Print-based	A printed text that combines different modes.	Graphic novels, posters, magazines, leaflets etc.
Transmedia	This text type employs a variety of media channels to tell a story or convey a message (each of which uses various modes). A multimodal text inside another multimodal text is called a transmedia text.	Role-playing activity games, interactive mobile games etc.

Linear vs non-linear

When reading a novel, the normal way a reader would approach the text is by following an order or using a linear method, i.e. starting from the cover, you would proceed to the beginning of the story until the end. But the multimodal texts of today are often non-linear in organisation and are more layered. In order to change the reader's engagement with a text, non-linear media texts are frequently complex with various information and integrated systems. To lead the reader in a certain direction, an online blog, for instance, would make use of written language as well as pictures, videos and carefully curated layouts to achieve a desired outcome.

Analysing multimodal texts

To analyse a multimodal text, you must first be able to recognise the various modes and the context in which they are used. The next step is to identify those modes and use them as evidence to support your claim(s). You should go into great detail describing these modes too. Recognising the various modes in a multimodal text and using them to support your arguments are important parts of the analysis process.