«Using a social semiotic approach to multimodality: researching learning in schools, museums and hospitals Jeff Bezemer, Sophia Diamantopoulou, Carey ...»
National Centre for Research Methods Working Paper
Using a social semiotic approach to
multimodality: researching learning in schools,
museums and hospitals
Jeff Bezemer, Sophia Diamantopoulou, Carey Jewitt,
Gunther Kress and Diane Mavers from MODE node
NCRM Working paper 01/12
Using a Social Semiotic Approach to
Multimodality: Researching Learning in Schools,
Museums and Hospitals
Jeff Bezemer, Sophia Diamantopoulou, Carey Jewitt, Gunther Kress and Diane
Abstract The aim of this paper is to show how a substantive area of social research –learning– can be investigated using a multimodal social semiotic approach. We apply the approach to three different institutions – a school, a museum and a hospital, illustrating key concepts and addressing issues around pedagogy and technology in contemporary society.
A multimodal social semiotic approach focuses on meaning-making, in all modes. It is a theoretical perspective that brings all socially organized resources that people use to make meaning into one descriptive and analytical domain. These resources include modes such as image, writing, gesture, gaze, speech, posture; and media such as screens, 3 D forms of various kinds, books, notes and notebooks. All of these modes and media are also used in environments designed for learning. That makes a multimodal social semiotic approach particularly apt for studying learning.
2. Multimodal designs for learning
3. Multimodal signs of learning
3.2. Bodily action
4. Historical comparisons of multimodal designs for learning
5. Implications: ‘Teaching’, ‘competence’ and ‘canonicity’
NCRM Working paper 01/12
1. Introduction The paper illustrates three ways in which a social semiotic perspective on multimodality can illuminate learning. First, it shows how ‘educators’ represent the world and establish pedagogic relations through multimodal designs for learning. We explore this in Section 2, where we discuss an excerpt from a textbook for secondary schools, a museum exhibition, and a demonstration in a science lesson in a primary school. Second, a social semiotic approach to multimodality draws attention multimodal signs of learning. This is the focus of Section 3. Here we look at signs of learning in the drawings and in the bodily actions of learners: drawings made by some of the visitors of the museum exhibition and some of the students in the science classroom (Section 3.1);
and the bodily actions of a medical student who assists in a surgical operation (Section 3.2). Third, a social semiotic approach to multimodality enables researchers to investigate social, pedagogic, and technological change. We illustrate this in Section 4 by comparing multimodal designs for learning in the early and mid-2000s in a secondary school classroom in London. In the concluding section of the paper we revisit some ‘old’ notions related to learning -‘explication’, ‘canonicity’, and ‘competence’- to further illustrate a multimodal social semiotic lens.
The paper is itself pedagogically framed. We aim to sketch out a theoretical lens, a social semiotic ‘gaze’, introducing key concepts (listed in Box 1) and using data excerpts taken from a number of different research projects. We do not discuss the (qualitative) methods of data collection and analysis used in the projects from which the examples are drawn. References to project reports can be found at the end of the paper, and future MODE working papers will introduce a number of different multimodal methods. One methodological point we would like to make is that the approach presented here brings together different materials as sources of evidence, ranging from drawings to video recordings, from photographs to field notes, and it seeks to develop semiotic categories that enable us to understand the differences as well as the similarities between these materials.
Box 1: Key concepts in Multimodal Social Semiotic Research on Learning Mode Medium Affordance Interest Ensemble Designs for learning Reading path Curriculum Sign of learning Transformative engagement Canonicity Competence
2. Multimodal designs for learning In all communication, in all domains of the contemporary social world, meanings are made in ensembles drawing on and consisting of different modes: with gestures and speech, with objects, in writing, with images, gaze, through posture, and actions of other kinds all contributing meaning;
always with several of these orchestrated in complex conjunctions. Each of the modes in such ensembles offers specific affordances, that is, potentials for communication. As a quite usual example, consider Figure 1.
Figure 1: Excerpt from a Science textbook (Science Education Group 2002, p. 90)
Figure 1 is an excerpt from a Science textbook for secondary education in England. Here, writing provides a description of the processes and entities involved in digestion: taking in, eating, breaking down, molecules, energy, et cetera. Image provides a depiction of the shape, size and placement of the organs involved in those processes. Without the use of either the one or the other, the information provided by the written account or the image alone would be severely limited, relative to the information which the curriculum suggests is needed. That is one of the key premises of multimodal social semiotics; that meaning-makers always draw on a multiplicity of modes to make meaning. These modes are put together, organized, arranged, into a multimodal design. The makers of the text in Figure 1 have used writing and image, but also typography, and layout, as modes of representation. Selections are made in each mode (we just mentioned the processes that described and the shapes that are depicted); and each mode offers ways to highlight that to which the learner’s attention is to be drawn. For instance, in typography, the size and weight of type is used. This is how the makers of a text can create reading paths and shape how learners navigate the text. In other words, the design is a sign of the interest of the ‘educator’ (used here metonymically to refer to all those involved in the making of the text, including authors and graphic designers).
Multimodal designs for learning can be found in the two-dimensional space available in a textbook, as well as in three-dimensional spaces such as classrooms and museums. To provide an example of the latter, we turn to the design of a museum exhibition. In museums, curatorial teams design environments for visitors to engage with. An exhibition, such as ‘London Before London’ at the Museum of London, presents arrangements and displays of ‘cultural objects’ - original archaeological artefacts as well as replicas, texts, labels, models, computer screens, sounds, lighting, reconstructions of prehistoric huts and panels of all kinds in space (see Figure 2).
NCRM Working paper 01/12
The design of the exhibition suggest a particular take on the world: how this world might have been and could be imagined, what is known of the past and ways in which it should be viewed in the present. In the exhibition, the left-hand side of the exhibition space for instance, consists of a continuous glass case filled with objects with captions, against a plain background flooded by blue lighting. Many of the objects displayed were found in the Thames, and the choice of lighting suggests a relation of the objects to the river. However, the manner of display - in large glass vitrines, as well as the lighting - lean heavily on or reproduce an aesthetic much more familiar from an art gallery. This indicates the curators’ interest to endow the objects with the status of works of art, using resources of display more usually found in an art gallery.
Figure 2: Image from the 'London Before London' exhibition
The right-hand side of this exhibition has wooden panels alongside some display cases. It is an arrangement dominated by writing, with suggestions of environmentalist, spiritual and poetic genres introducing an aesthetic discourse. At the same time here too there is an interest to ground the information scientifically through graphs, bringing ‘scientific evidence’ from various disciplines and with ‘scientific discourse’ as a framing in some texts. These overarching discourses aesthetic, environmentalist, scientific and mixtures of these - regulate the design and reflect the interests of meaning-makers in the design team, their epistemological positions, as well as larger level museum policies. The exhibition resonates with an overall interest in foregrounding the prehistoric artefacts, together challenging current perceptions about their importance and that of the period.
As in the textbook, the multimodal ensembles and their spatial disposition suggest ‘reading’ paths for the visitors, via organising principles for constructing salience in the various modes, including that of layout. Criterial aspects of the representations, such as colour, size, angle, and position have an effect on the choices the visitors are encouraged to make in navigating the space, as well as on how they might ‘accommodate’ their interests through what is made available. Similar observations can be made about the textbook example. Both examples show how a body of knowledge, a ‘curriculum’, in educational terms, is articulated multimodally, using the resources available to the educator. Our next example shows how other sites, such as classrooms, make available a different set of resources.
The example was taken from a primary classroom in London. It features a teacher and his class of students aged 7 and 8 years. He articulates a part of the science curriculum –“forces”– by reading out the unit- and lesson-objectives from a screen, asking questions, enacting processes, making a mind map, which is copied by the class, and showing different kinds of magnets. He uses a range of modes for communicating at the same time: placement of 3D objects, action on and gesture above the objects, and speech (see the multimodal transcript in Figure 3). With this multimodal ensemble the teacher establishes ‘experimental conditions’ and invites the class to predict what would happen “if I move them [i.e. the magnets] closer together”. Choosing to use the ‘visualiser’ in the classroom – a digital display technology that magnifies whatever appears in view of a video recorder on the whole-class screen – was a way of directing the attention of the class.
Figure 3: A multimodal transcript of a demonstration in a science lesson The three examples discussed so far show not only that the different sites –textbook, museum, classroom- make different sets of modes available, they also show that this leads to different potentials for what might be learnt (‘curriculum’) and what involvement there is of learner and educator in the production of a school subject. ‘Pedagogy’, in this approach, is the transposition of social relations from the social world around the school into the classroom, as a metaphor of what kinds of social relations can be imagined, and which the school might prefer.
3. Multimodal signs of learning In this section we explore how learners respond to the multimodal designs for learning discussed above. We shift attention from the interest and design work of ‘educators’ to the learner’s displays of engagement. We focus on learners’ multimodal signs of learning in their drawings (Section 3.1) and bodily actions (Section 3.2).
Before we look at drawings made in schools and museums we consider an instance of learning in an environment which was framed and constrained somewhat differently. A three year old, sitting
on his father’s lap, draws a series of circles, seven to be exact (see Figure 4). At the end he says:
“this is a car”.
Figure 4: Drawing by a 3-year-old child Whether from the perspective of learning or of meaning-making, the question arises as to how this is or could be “a car”. While drawing, he had said “here’s a wheel, here’s another wheel, that’s a funny wheel…. This is a car”. In other words, for him the criterial feature of a car was its ‘wheelness’, that it had (many) wheels. Wheels were represented by circles; and ‘car’ was represented by the arrangement of seven circles. To represent wheels by circles rests on a process of analogy: wheels are like circles. The result of the analogy is a metaphor; similarly with the representation of car - ‘a car is many wheels’. The meaning made here is a succession of two metaphors: wheels are (like) circles; and many circles are (like) a car.
We might ask further why, for this child, wheels could be the criterial feature for ‘car’. If we imagine the eye-level of a three year old, looking at the family car (in this case a 1982 VW Golf, with its prominently visible wheels, especially at the observer’s height) we might conclude that this meaning-maker’s position in the world, literally, physically, but also psychically, affectively, might well lead him to see cars in that way. His drawing therefore represents his ‘position’, his ‘interest’, arising out of his (physical, affective, cultural, social) position in the world at that moment, vis-à-vis the object to be represented. From the perspective of learning we can say that his interest shapes his attention to a part of the world and, in this, acts as the motivation for principles of selection.
Our point is that it is the interest (in the sense just given) of the meaning-maker which determines what is taken as criterial about an entity by her or him at the moment of its representation. The child’s drawing suggests a view of the world that is historically, socially and culturally shaped.
What the meaning-maker takes as criterial then determines what (s)he will represent about that entity. Only what is criterial is represented; other features are left out or are backgrounded. Hence representation is always partial. The drawing is the result of the child’s work in his engagement with the world, embodying his (distinctly different) interests.