«Using a social semiotic approach to multimodality: researching learning in schools, museums and hospitals Jeff Bezemer, Sophia Diamantopoulou, Carey ...»
The teacher scanned student responses to the poem, including students’ own poems, and displayed these immediately on the Interactive Whiteboard (IWB). The student texts displayed on the IWB became objects for discussion that both the teacher and fellow students manipulated and annotated. A shared, malleable text was created that opened up new pedagogic possibilities that could affect the configuration of authorship and authority in the classroom. The teacher’s annotation and marking of the student texts on the IWB has transformed what had been, usually, a semi-private activity into a public one. This makes both the criteria and the process for assessment explicit.
The sense of what can and needs to be displayed has changed in the time between the two lessons, as have the technologies of display. In 2000, writing and speech were in the foreground;
by 2005, however, image, colour and layout have, alongside writing, become central to the pedagogic resources of the classroom. The changing semiotic landscape of the classroom has an effect on the curriculum and the pedagogic function of texts – what texts are presented, how texts are presented, and what can be done with them. In the lesson from 2000, the use of the OHP supported the display of the written poem with line numbers, as a photocopy from a book. By 2005, the poem had become integrated with images downloaded from the Internet across several slides of a teacher-made PowerPoint.
More generally, comparison of the two data-sets, and observations of school English in 2009, suggest that changes in the relationship between image, speech and writing have been extended and embedded in the English classroom. It is now common for English teachers (although there are likely to be generational differences in this) to show a clip of digital video (often via Youtube) or to display an image – often downloaded from the Internet – to offer a route into a ‘concept’.
Teachers frequently use PowerPoint presentations to present their argument, they annotate texts visually or they connect to a webpage. The use of image is also prevalent in students’ work in English, with the use of clipart, digital photographs – taken by students or downloaded from the Internet – designed as PowerPoint presentations and project work, both in class and out of school for homework.
This has begun to reshape the work of the teacher and the student. The ‘contemporary teacher’ is involved in the pedagogic design of digital multimodal texts that were rarely seen in 2000. The student analysis of written ‘imagery’ in poems is now often (re-)mediated by actual images – indeed in this classroom the teacher began the discussion of ‘concepts’ by showing images downloaded from the internet. What is to be learned and how it is to be learned is being reshaped by the multimodal potentials of digital technologies used by teacher and student uses.
This prompts the question: what are the social and educational implications – the gains and losses of this process?
One significant difference for textual analysis is that the starting point for the introduction and the analysis of the poems is different. This difference appears to be underpinned by changes in the use and function of writing, speech and image in the classroom. The starting point for textual analysis in the lesson from 2000 was provided by a whole-class discussion of the poem’s title and the students’ use of the dictionary to ‘look up’ words in the title (and in the rest of the poem). The dictionary was the irrefutable, taken for granted reference point and authority. In 2005, the starting point for textual analysis lesson was a whole-class discussion of the image accompanying the poem displayed on the IWB and ‘brainstorming activity’ engaged in by the whole-class. The role of the dictionary in textual analysis had changed, from its central position in the 2000. Indeed, where in 2000 there were two or three copies of the larger Oxford English Dictionary on every table, by 2005 there was no copy of the dictionary in the classroom. By 2005, the meaning of words had become ‘anchored’ and defined through images downloaded from the Internet that the student were asked to match to words in the poem, such as ‘congregation’.
Our comparison suggests a broad move towards ‘capturing’ and displaying the work and opinions of students: a move from talk as ephemeral to the concretised display of talk. In 2000 there was a firmer boundary between the work of reading the poem and of analysing the poem than is the case in 2005. In 2000, the poem was read aloud twice before analysis for meaning began; in 2005 analysis for meaning begins with an engagement with the image before the written text of the poem is even introduced. The potentials for meaning made possible by changes in the sociotechnological environment of the classroom raise quite new possibilities for decisions for teachers and students, with far-reaching implications for curriculum and pedagogy, and student identities.
5. Implications: ‘Teaching’, ‘competence’ and ‘canonicity’ To conclude this paper we discuss some of the wider implications of a multimodal social semiotic approach to researching learning. Three terms are in focus here: ‘teaching’ (which relates back to Section 2 on multimodal designs for learning), ‘competence’ (which relates back to Section 3 on signs of learning) and ‘canonicity’ (relating back to Section 4 on historical changes).
‘Teaching’ ‘Teaching’ is often defined as ‘making explicit’ the ‘unspoken’, ‘practical’ knowledge of experts. The examples discussed in Section 2 show that explication is not the exclusive domain of the modes of speech and writing. Indeed, in many contexts, image, or gesture, are modes better suited to ‘make explicit’, or even the only modes available to make anything explicit at all. Thus we see educators using different modes in different contexts to make explicit what needs to be learnt. For instance, in writing and speech science educators can express the processes involved in digestion exceedingly well, but when the forces involved in magnets are discussed in the classroom gesture becomes the preferred route to knowing. Multimodal social semiotics, in turn, aims to document all of these forms of explications so as to provide an inclusive picture of learning across different sites. The historical comparison in Section 4 also pointed to social and technological changes within one particular site, the classroom, and its effects on the role of the educator. While there has at times been some concern that digital technologies might do away with the central role of the teacher, the implication of the analysis here is rather different. It suggests that the role of the teacher, far from being ‘de-professionalized’, is becoming one of the teacher as rhetor and designer of different sites as maximally effective environments for learning.
‘Competence’ The examples presented in Section 3 suggest that we can potentially recognize signs of learning in any mode, or combination of modes, on any timescale, and in any site. At the moment, only a small selection of these is recognized in assessments. We acknowledge that cultures and societies do recognize these signs to different degrees, privileging one above the other, or treating one as ‘richer’, ‘better’ than the other (for instance, in ‘mainstream’ secondary school curricula in the Western world writing skills take precedence over drawing skills).
NCRM Working paper 01/12
Multimodal social semiotics assumes that power relations are manifest in all forms of recognition;
yet rather than establishing an opposing hierarchy of valuation, it sets out to investigate how people use and continue to develop modes of communication in response to social and cultural demands. Thus, a multimodal perspective draws attention to that which is not (yet) ‘curricularized’.
For instance, part of learning is understanding the apt resources to bring to bear in a given context to make meanings and then to express meaning: on one occasion a verbal response to the teacher is expected, on another, learners are expected to display their engagement through modes other than speech – gaze and posture, for instance. This holds true for all meaning-makers we observed, whether as teachers and learners in schools or as an experienced professional teaching a relatively inexperienced member of a profession at work.
The changing semiotic landscape also poses new questions about what is learnt. For instance, where up to two decades ago maybe, competence in relation to one mode, writing (‘literacy’), was seen as sufficient for the task of composition of text, we now need to understand the semiotic potentials of all modes involved in text making. Now, when text consists of image and writing say, specific forms of textual cohesion and coherence emerge and theoretical means are needed for making sense of these. Where previously grooved routines of convention could serve as reliable guides in composition, in a multimodal world there is a need to assess on each occasion of textmaking what the social relations with an audience are, what resources there are for making the text, what media are going to be used, and how these fit with what is to be communicated and with a clear understanding of the characteristics of the audience.
‘Canonicity’ One of the semiotic changes we identified in Section 4 is the shifting relationship between image and writing in designs for learning. The visual is no longer – if indeed it ever was – an illustrative adjunct to word; images are used fully in representation; they are integrated in multimodal ensembles. This move speaks of the need to make curriculum knowledge “relevant” by connecting with students’ out-of-school experience; the desire to increase student “engagement” through “interactivity”; as well as the pressures of examination and the promise of ‘pace’, ‘speed’.
Increasingly, images now provide the starting point for an English lesson.
This has far-reaching effects for learning: through the texts that come into the classroom, how they are mobilized, how they circulate and are inserted into social interactions. This changes the place, the functions and uses of image, writing and speech. The boundaries between canonical texts and the texts of the everyday, of the aesthetically and historically valued, of the mundane and the canonical are changed. These changes mark the social and political boundaries of English – determined by teachers, schools, Local Education Authorities, by policy and by diverse social interests – boundaries hitherto tightly guarded and regulated by a highly prescriptive policy context. Drawing texts from the Internet (for example, from image banks or Youtube) connects English with the experiences and technologies of the ‘out-of-school’ in ways that question the boundaries of canonical knowledge and what counts as socially valued. This changes the semiotic landscape of the English classroom, even though these changes vary across an uneven social terrain.
Acknowledgements This paper was written as part of MODE, a research and research training programme based at the Institute of Education, University of London and funded by the Economic and Social Research Council. See mode.ioe.ac.uk. The textbook example draws on a project by Jeff Bezemer and Gunther Kress, ‘Gains and Losses’, on changes in representation in textbooks. This work was funded by the Economic and Social Research Council. See Bezemer & Kress, 2008. The ‘magnets example’ draws on a project by Diane Mavers on the use of digital visualisers in the primary school classroom. This work was funded by the Centre for Excellence in Work-Based Learning (WLE) at the Institute of Education, University of London. See Mavers, 2009; Mavers, 2011. The museum examples are drawn from a project on museum visitors, ‘The museum, the exhibitions and the visitors’ by Sophia Diamantopoulou, Gunther Kress, Staffan Selander and others. This work was funded by the Swedish Research Council. See Diamantopoulou & Kress, forthcoming). The historical perspective on the English classroom was illustrated with examples from two different projects. The first example is from the School English Project by Carey Jewitt, Gunther Kress and others. See Kress et al., 2005.
NCRM Working paper 01/12
The second example is from the London Challenge project by Gemma Moss and Carey Jewitt.
See Moss et al. 2007. Both projects were funded by the Economic and Social Research Council.
The surgical examples are drawn from the ‘Mapping Educational Activity in the Operating Theatre’ project by Roger Kneebone, Gunther Kress, Jeff Bezemer and Alexandra Cope. This work was funded by the Royal College of Surgeons and the London Deanery. See Bezemer et al., 2012.
Further reading References to publications on the examples used in this paper are given in the acknowledgements section above. For general introductions to multimodal social semiotics (i..e. not focused on learning per se), please see Hodge & Kress, 1988; Kress & van Leeuwen, 2001; van Leeuwen, 2004; Kress, 2010. For introductions to multimodal methods in social semiotics, please see Bezemer & Jewitt, 2010. For a general overview of multimodal research on learning (i.e. not only social semiotic studies), see Jewitt, 2008. For a general overview of multimodal research, please see Jewitt, 2009.
References Bezemer J, and Jewitt C. 2010. Multimodal Analysis: Key Issues. In: Litosseliti L, editor. Research Methods in Linguistics. London: Continuum.
Bezemer, J. and Kress, G. 2008. Writing in Multimodal Texts: a Social Semiotic Account of Designs for Learning. Written Communication 25, 2, 166-195 Bezemer, J., G. Kress, A. Cope & R. Kneebone (2011). Learning in the Operating Theatre: A Social Semiotic Perspective. In V. Cook, C. Daly & M. Newman. Innovative Approaches to Exploring Learning in and through Clinical Practice. Abingdon: Radcliffe.
Diamantopoulou,S., and Kress, G. forthcoming. Museum visitors’ designs and signs of learning: A multimodal approach in: Diamantopoulou, S, Insulander, E., Lindstrand, F. (eds), The museum, the exhibition and the visitor: Politics, design and learning.
Hodge B, and Kress G. 1988. Social Semiotics. Cambridge: Polity.