RELAZIONE

THE MCA SYSTEM: ITS ROLE IN THE DEVELOPMENT OF MULTIMODAL CORPUS LINGUISTICS (09/10/2004)



Anthony Baldry,
University of Pavia

Michele Beltrami,
In4mik, Castelvetro Piacentino (PC), Italy

1. Introduction

This paper describes MCA (Multimodal Corpus Authoring System), a web-based multimodal concordancer, whose Home Page (http://mca.unipv.it) is shown in Figure 1. This system allows users, such as University teachers, researchers and students, to analyse film texts using Windows Internet Explorer 6 and Windows Media Player series 9 (or higher). The paper may be read in conjunction with Basics, MCA’s online presentation, which can be found by clicking on About MCA on MCA’s Home Page (Figure 1, Callout 2) and then on Overview: (Figure 2, Callout 1). The MCA tool is used in, and has developed from, research undertaken within three Italian inter-University COFIN projects, Citatal, Linguatel and Didactas, into the meaning-making processes and meaning-making structures that characterise film texts (Baldry 2000a, 2000b, 2002a, 2002b, 2003, 2004, Thibault 2000, Taylor Torsello, Brunetti, Penello 2001, Taylor Torsello, Maria Grazia Busà, Sara Gesuato, in press). More specific information about MCA and a general overview of research into multimodality and multimodal corpus linguistics can be obtained by clicking Overview: Bibliography.


Figure 1: Multimodal Corpus Authoring System: Home Page

The definition of film text in this research goes beyond the notion of feature films and embraces a wide variety of film types: feature films, news and sports programmes, cartoons, documentaries, advertisements as well as videotape and digital recordings of lectures, public meetings, conferences, weddings, holidays, children studying at school and many others. It also embraces very different types of audiences: films for the general public (cinema, DVD, TV or web-based films), those destined for more restricted audiences (company training films, recordings of University lectures, recordings of children telling stories) as well as those produced exclusively for research purposes (such as films of websites) (Baldry in press a). All of these text types enact meaning-making processes and create ties with their audiences which differ from those found in language-only texts in more ways than one (Baldry 2000a, 2000b, Thibault 2000). For the purposes of this research, film texts may thus be defined, without regard to the particular channel of communication used to show them, as specific instances of any type of film media, which, whether projected in cinemas, broadcast on TV, distributed as home video or displayed through the Internet, and quite unlike other texts, unfold, frame by frame, at a fixed speed, displaying different and constantly-varying interplays between semiotic resources such as ambient sounds, images, gestures, music and language. More technically speaking, such texts are both dynamic and multimodal in nature (Thibault 2000).



Figure 2: The “About MCA” Help Section showing the MCA Login Page


2. MCA and its place in corpus-based film text analysis

Why do we need to analyse film texts and why do we need MCA? In answer to these questions, we may note that, whether as feature films, cartoons, advertisements, lectures or documentaries, film texts change our outlook on society. Film text analysis consequently needs to provide an account of the way in which film texts function in society and to characterise the mutual interdependence between film texts and the changes that take place in society. In this respect, changes in social organisation, technological developments included, affect the way in which films are made and, more significantly, the way they function as semiotic objects. As well as accounting for the role film texts play in society, film text analysis also needs to provide an account of the mechanisms that allow these types of text to function as meaning-making units (Lemke 1998, Thibault 2000, Baldry 2000a, 2000b, in press a). Film text analysis typically examines the salient features of film texts such as the synchronisation between the meaning-making resources deployed in film texts (Thibault 2000, Baldry, Thibault 2001, in press a, in press b). This is a crucial aspect of the meaning-making process in film texts which becomes apparent when, for example, films are dubbed or subtitled (Baldry, 2002a, 2002b, Taylor 1999) or when, as illustrated below, there is a need in both text analysis and in classroom concordancing to link film scripts, line by line, to the corresponding film sequences (Baldry, Caldirola, Grunther in press). Because of these differences between language-only and multimodal texts, some of the goals of multimodal corpus linguistics (Baldry, 2000a: 36-37, 2000b: 81-85) are rather different from those of lemma-based concordancing (Baldry, Thibault 2001, in press a, in press b). In particular, the focus in film text analysis is on the identification of the units of meaning that make up these texts such as phases (see below). Indeed, unlike traditional lemma-based concordancing, where the focus is often on recurrent lexicogrammatical patterns, such as collocations and colligations in very large and hence very reliable corpora (Aston 1997), the focus in film text analysis is instead on textual characteristics often with a view to understanding the differences and similarities between genres. One interpretation of the concern with textual characteristics, the one underlying this paper, lies with the study of phases (Gregory 1995, 2002, Thibault 2000, Martinec 1996, 1998) and phase types within multimodal corpus linguistics. It may, in part, be seen as relating to efforts to reconstruct the generic structure of film genres in keeping with Hasan’s studies of generic structure potential (Hasan 1984, 1996, Baldry in press c, Baldry, Thibault in press b).
During the initial design work on MCA it quickly became apparent that the divide between film texts and non-film texts is not always as clear-cut as it might at first seem (Baldry 2000b: 85). In modern society, websites, for example, have become an increasingly significant semiotic medium, typically characterised by constant juxtapositions of written discourse, visual artefacts, embedded film material and animations. The cartoon-like people, animals and objects present in websites resemble the typical static multimodal clusters of the printed page until, that is, when selected by a mouse, they are transformed into dynamic elements, typical of a TV cartoon. This may be disconcerting for those who like to see text types slotting into clearly-defined pigeon holes but scholars working on genre theory and the evolution of specific genres in English need to be able to construct mixed-media corpora that help them understand how genres are affected by different types of media and thus frequently need to cross, and indeed redefine, well-established boundaries. An example of this is the Recipe corpus constructed by one of the authors (Baldry). This is a mixed-media corpus containing recipes taken from TV cookery programs, websites, radio broadcasts, cookbooks and food packaging and thus relating to combinations of written, oral and multimodal discourse (see below, see also Baldry in press c). A similar film text/non-film text relationship arises as mentioned above, where there are strong intertextual ties between texts, such as those existing between a film and a film script (see below and Baldry, Caldirola, Grunther in press). So we can legitimately draw the conclusion that concordancers need to be built in such a way as to bridge the divides existing between film texts, non-film texts and those texts, such as web pages, which share features of both. It is not good for scholarship to be in a situation where, within the same research or teaching project, film texts have to be analysed with one type of concordancing instrument, written language-only texts by a second set of concordancing tools and oral discourse texts by yet a third.
The basic problem for film text analysis is, however, that a single concordancing system for all types of discourse remains a pipedream. Or rather this was the case until MCA was built. Scholars in the field of film text analysis were in a stone age in which they had to make their own tools and develop their own techniques ex-novo. Indeed, to the authors’ knowledge, there is still no commercial software tool, or indeed, for that matter, any other software tool that applies an in vivo approach to the annotation, retrieval, viewing and comparison of films or which, thanks to the use of streaming video files (in MCA the *.wmv format), allows the concordancing of film texts to go online. Whereas software-assisted techniques have long been established in many branches of text studies as the ‘pots and pans’ of the trade, the same is still not true film text analysis though MCA may hopefully contribute to changing this position.
What is meant by an in vivo approach? In answer to this question we need to clarify the two basic methods through which the meaning-making processes of film texts can be analysed: the in vitro approach and the in vivo approach (Baldry in press c; Baldry, Taylor 2004). Both have advantages and disadvantages; both have reached different stages of development.
An in vitro approach is based on the idea of slicing up a film into frames and reconstituting them in various ways and led to the development of the static variety of multimodal transcription. Thus, in the mid-1990’s, one of the authors (Baldry) experimented this approach in relation to videorecordings of children using the postproduction facilities available in Adobe Premiere to import sequences of stills into a Word file table in which the columns and rows were annotated. All this has been reported both in publications (Baldry, 2000a, 2000b in press b) and in degree theses (Lidgley 1998, Palumbo 2000). It led to the subsequent development of a more complete form of multimodal transcription by Baldry and Thibault, which attempts to reconstruct the meaning-making processes in specific film texts by linking sequences of individual frames to systematic semiotic descriptions of these texts (Baldry 2000a, 2000b, in press a, in press b, in press c, Thibault 2000, Baldry, Thibault 2001, in press a, in press b) based on the systemic-functional approach developed by Halliday (Halliday 1978, 1989a, 1989b, 1994) and co-workers, and taking Gregory’s work on phasal analysis into particular account (Gregory 1995, 2002). One of the very first public presentations of the multimodal transcription was presented by Baldry in a workshop at the 25th ISFC held at the University of Cardiff (Taylor, Baldry 2001a) in which Gregory expressed considerable interest in the application of his phase-and-transition approach to textual analysis to film texts. Despite the very great limitations of the in vitro approach, that have been pointed out since their inception (Baldry 2000a: 36, 2000b: 81), multimodal transcriptions have nevertheless been of considerable benefit to various scholars in English linguistics. In Italy, for example, it was subsequently adopted in various publications and research projects (Lidgley 2000: 180-182, Lombardo 2001, Taylor 2003, Vasta 2001, Taylor, Baldry 2001a, 2001b, Zago 2002). Further variants of this type of transcription are where two film texts are compared side by side (Baldry 2000b: 68-69), where phase types are included (Baldry 2004) and where the transcription is the basis for teaching and learning activity (Baldry 2000b: 82-83). The fact that multimodal transcriptions take on different forms and carry out functions has been repeatedly pointed out and is an indication at least in one sense of the flexibility of this tool.
There is no shortage of commercial software instruments that allow an in vitro approach to be applied when a static type of multimodal transcription is being produced (Thibault 2000: 311-385), including transcriptions concerned with frame-and-phase analysis of documentaries and lectures (Baldry 2000b: 67-69, 78-79). The matter is already different when a dynamic, corpus-based type of in vitro approach to multimodal transcription is envisaged in which the individual rows of a transcription, each consisting of a still picture plus annotations, are retrieved from a database. Indeed, the origins of MCA go back to the 1998-2000 period when a series of prototypes based on relational databases were developed by the authors including the direct forerunner of MCA, the Dynamic Transcription tool, a mock-up of which was shown at the 26th ISFC (International Systemics Functional Congress) organised by the National University of Singapore, followed shortly after by a fully-functioning prototype demonstrated at the 27th ISFC held at the University of Melbourne in July 2000 and at an invited seminar at the University of Birmingham in October 2000, only a matter of months before the first prototype of MCA was presented in a Linguatel Seminar in the University of Pavia in December 2000. In keeping with the dynamic principle, the Dynamic Transcription system (Baldry 2000a: 31, 2000b: 81-5) allowed multimodal transcriptions to be produced dynamically from rows of annotated film stills. These are more selective than a static multimodal transcription in that they retrieve recurrent features to the exclusion of others (Baldry, Thibault in press b). Dynamic multimodal transcriptions are thus created on the fly from a database but can be pasted into documents in such a way that they are effectively converted into the tables of rows reminiscent of static transcriptions. Even if they still fall within the domain of the in vitro approach, this type of transcription allows researchers to take the very first steps down the corpus-based trail concerned with the identification of recurrent meaning-making units and reading paths in film texts that leads to the discovery of meaning-making patterns (Baldry, Thibault 2001, in press a, in press b). Nevertheless, concordancers like Dynamic Transcription, now incorporated within MCA, have to be built ex novo since, to the authors’ knowledge, there is no commercial software tool that does this job in a systematic way and which reconciles the need to shift constantly between dynamic and static interpretations of the multimodal transcription.
The in vitro approach, in its various manifestations, is undoubtedly extremely useful to scholars. However, film texts can also be analysed in an in vivo format, an approach that avoids the ‘dissection’ of a film text into frames and concomitant loss of meaning-making resources and processes entailed by the in vitro method. This method allows a user to annotate film texts and subsequently view and compare all the sequences that share similar annotations not as a sequence of stills but instead as film. This ‘living text’ approach requires software tools capable of linking sets of annotations to segments of films in order to produce concordances, each of which is directly linked to a viewable film sequence. However, the in vivo approach does not, at least in principle, exclude other possibilities. The guiding philosophy with the MCA project has been to make every effort, however daunting the task may be, to embrace, as discussed below, all possible types of concordancing, thereby reconciling: (1) in vitro and in vivo approaches to film text analysis (2) meaning-oriented concordancing with more traditional lemma-based concordancing and (3) concordancing that takes in account a wide variety of text types, each displaying various combinations of dynamicity and multimodality.
The methods described above were used, in conjunction with others, in papers and workshops at the 28th ISFC held at Carleton University Ottawa in July 2001. In the Multimedia Semiotic Analysis – How Language Functions in Partnership workshop, Lemke discussed websites in terms of reading paths called ‘traversals’ (Lemke 1998) while Thibault spoke of the integration of semiotic resources in videotexts. In a paper entitled Theory and practice: The analysis of multimodal texts, O’Halloran presented an analysis of a scene from Polanski’s film Chinatown reconstructing gaze and choices from other metafunctionally-based systems using Adobe Premiere overlays as the basis for multimodal transcription (O’Halloran 2004). Another workshop ‘Multimodal Corpora: Theory and Application’ (Baldry, Thibault, Taylor) was concerned with the techniques by which multimodal corpora can be annotated. In this respect, Taylor illustrated the characteristics of documentaries (Taylor 2003), Baldry discussed annotational techniques used to link written and oral slogans, logos and other visual and verbal mini-genres found in the end phase of car advertisements, Thibault discussed visual transitivity frames and tagging in TV car advertisements. Much of the research presented in this workshop represented a further development of work originally presented by Thibault in the Linguatel Seminar in Pavia in December 2000 and by Baldry at ESSE 5 in the University of Helsinki 2000 Congress (see Baldry, Thibault 2001, in press a, in press b, Aston, Burnard 2001).
Following the demonstration of an MCA prototype in 2000, MCA 1 was first demonstrated publicly by Baldry and Taylor at the First International Conference on Multimodality organised in Salzburg in January 2002 (Ventola, Charles, Kaltenbacher in press). The original version focused particularly on the annotational tools described in Section 3. It was quickly followed by MCA 2, which was illustrated at the LREC Congress in May 2002 at Las Palmas by Baldry, Beltrami and Taylor (Baldry, Taylor 2002) and at a workshop organised by Baldry, Kantz, O’Halloran, Taylor and Thibault at the 29th ISFC held at the University of Liverpool in July 2002. Among other refinements (Baldry in press c), this release made MCA more relevant to online classroom concordancing (see Section 4). Subsequently, a more fully-fledged release was shown by Baldry and Thibault in the BAAL/CUP Seminar on Multimodality and Applied Linguistics in Reading July 2003. MCA 3, with its security management system was first demonstrated at the Didactas Seminar held at the University of Trieste in January 2004. This version added a security system which enabled users to construct corpora with a greater guarantees of privacy (see Section 5). MCA is not a completed project. The current release (3.4.7.7, September 2004) will be superseded by MCA 4 already in the advanced planning stage, which should allow further steps to be taken on the road to ensuring improved integration between the various approaches to concordancing described above and illustrated in more detail below.

3. MCA as an annotational tool for film texts

MCA has adapted concordancing techniques developed in relation to language-only texts (Sinclair 1991, Biber, Conrad, Reppen 1998, Partington 1998) so as to make them applicable to film texts in a way that satisfies a user’s quest for patterns in film texts (Baldry, Thìbault 2001, in press a, in press b, Baldry, Taylor 2004, Baldry, Thibault in press a, in press b, Taylor Torsello, Baldry in press). So far, this quest has been based on the comparison of texts and their component parts and includes, for example, assessments of the significance of variations in rhythm as a source of meaning in film texts (Martinec 1996, Thibault 2000, Baldry 2000a, 2000b, in press b). It also includes the ways in which soundtracks integrate with videotracks (Baldry 2004) within the different patterns of phasal and metafunctional organisation of texts (Baldry 2004, Baldry, Thibault in press a, in press b).
A manual approach to tagging is adopted in MCA as with other systems such as Systemics (O’Halloran 2003). There are, in fact, three main annotational tools in MCA 3 each presented on a dedicated page: Media indexing; Grammar definition and selection; Sequence analysis. These tools allow authorised users to segment film into functional units and, while viewing them, type out annotations relating to patterns of meaning enacted by the various semiotic resources deployed within each unit. A fourth tool, Task Definition, deals only marginally with annotation, since its main concern is with ancillary and support documents, including teaching materials within the strand of research into online concordancing that is concerned with document management. However, other additional forms of annotation, for example, for subtitling and captioning, are planned for subsequent releases of MCA.


Figure 3: The Project Page: the first step in building a corpus

These annotational tools can only be used once a Project has been set up. An example of a Project is the Recipe corpus currently under development that views recipes in a variety of film formats e.g. as parts of TV cookery programmes but also as filmed websites. The Project Page relating to this corpus is shown in Fig. 3. Like all the pages in MCA, the Project Page contains a Tool Box, a group of buttons providing access to the various pages and functions. The contents of the Tool Box vary according to the page being displayed and the user’s authorisation level (see Section 5 below).
As Figure 4 shows, the first set of annotational tools in MCA can be found on the Media Indexing Page. They carry out three basic functions: splitting films (in the *.wmv file format) into virtual sequences of any length (Figure 4, Callouts 5, 6 and 7) naming the individual texts and their parts and, finally, where appropriate, providing a preliminary indication of their classification into text types. When retrieved through the MCA Search Page, their identity will thus be immediately clear. This extends to non-video format files such as radio broadcasts where the division into subtexts or sequences can be incorporated into the visual display (Figure 4, Callout 4). Vice-versa it also applies to no-sound texts i.e. those that only contain visual elements.


Figure 4: Media Indexing Page: the first annotational tool

Although to work correctly a *.wmv must contain both a soundtrack and a videotrack, either of these tracks can remain empty of content. Thus, from a semiotic, but not technological standpoint, the text either contains a soundtrack (e.g. a radio broadcast) or a videotrack (e.g. printed page corpora) or both (film in general). Moreover, using MCA’s tools, a *.wmv file can be cut up into units of any length, including individual frames and hence shown on a computer in their original size and form. In this way, texts such as radio broadcasts retain their original ‘non-video’ sound format, recipes from cookbooks keep their ‘look’ as printed single or double pages, while website recipes preserve their original ‘feel’ as static or animated pages since they can be viewed individually or in a sequence. The only difference is that, as compared with the original text, all these textual objects are now part of a single corpus that can be concordanced. The possibility of extending online analysis to texts which, at least in their original incarnation were not true film texts has been applied, in many MCA projects. In addition to the construction of mixed-media corpora such the Recipe corpus mentioned above, significant results have been achieved as regards the analysis of web pages which has helped to create a better understanding of the relationship between individual web pages and the websites and portals to which they belong (Baldry, in press b, Baldry, Thibault in press b).
As well as dividing the film up into sequences and providing each sequence with a name, the time-span instruments incorporated in the Media Indexing tool (Figure 4, Callout 7), also define a film sequence’s start and end with a precision running into millionths of a second. A sequence can thus consist of anything from a single frame to a complete film lasting for many hours. Parts of the same film text can be defined as belonging to one or more retrievable meaning-making blocks. In the case of the Recipe corpus, indexing has made it possible, for example, to retrieve the same recipe as a written text (typically a single frame), as an oral description (typically a summary description/discussion of the recipe in question), and as multimodal text (typically a cookery programme demonstration turning theory into practice).
As mentioned above, researchers often need to examine the individual phases and subphases in film texts. In this respect, it is as well to recall that a phase is defined as a meaning-making unit made up of diverse resources (movement, gaze, gesture, language, sound, etc.) that are co-deployed over a given stretch of time (Thibault 2000). In keeping with Halliday’s theory of metafunctions (Halliday 1978), the interplay between resources within a phase instantiates a specific combination of ideational, interpersonal and textual meanings (Thibault 2000, Baldry 2000a, 2000b, 2004). A phase is also a unit of action relating to a specific activity that carries out a specific function within a film text (Baldry 2004, in press a); longer phases are typically made up of subphases relating to specific activities such as walking, talking, sleeping or driving, which, when linked up in a sequence, constitute the phase’s composite meaning (Baldry 2004, in press b). Because phases carry out specific functions within a film text, it is possible to posit the existence of phase and subphase types within film texts (Baldry 2004, Baldry in press c, Baldry, Taylor 2004). Work to establish typical patterns of phasal organisation and typical functions of phases within film texts is being carried out using MCA within an approach to film analysis based on phase types (Baldry 2004). In the rigorously organised world of TV advertisements, for example, it seems that the phasal organisation of car drive advertisement subgenre (i.e. those which advertise cars by showing them being driven) has the following generic structure (Hasan 1984): Orientation Phase^Car-drive Phase^(Car-drive Phase)^End Phase in which the car-drive phase is usually made up of no more than three of the subphasal activities listed in Figure 11. Both at a phasal and a subphasal level some components can recur (indicated by round brackets), though recurrence is relatively rare because of time-limits operating in this genre. Where it does occur it is usually a marker of salience.
While, from a theoretical standpoint, some understanding of phasal structure in film texts has emerged, research into multimodality multimodal grammars for film texts are still in their infancy, though, in addition, to the work carried out by Baldry and Thibault, we should also recall multimodal approaches to the analysis of film carried out by O’Halloran (2004) and Iedema (2001) and in passing that multimodal grammars that are being developed in other fields, to give just one example, those for mathematical graphs and mathematical symbolism (O’Halloran 2005). Indeed, much of the research that lies behind MCA relates to gaining a better understanding of the characteristics that such a grammar might take. In MCA, a Grammar consists of descriptive labels relating to a set of semiotic choices, referred to variously as ‘labels’, ‘descriptors’ or ‘parameters’, that can be used to tag a corpus of multimodal texts. The grammars developed so far are mini-grammars based on very small samples and have the status of working hypotheses while attempts are made to understand the nature and kind of intersemiotic relations on which a multimodal grammar is based. The term ‘grammar’ is thus used in this paper in this restricted, exploratory sense. For example, a grammar may contain labels referring to the various phases present in a TV cookery programme: the announcement of the name of the dish being prepared, the reasons why it is being prepared such as the appropriateness to the season, age groups, diet regimes, the availability of the ingredients, instruments, methods, cooking times and so on. A single mini-grammar like this can be created in a short space of time in a classroom lesson, in particular one which involves half the class looking at the sequence adopted in a TV cookery programme and the other half working out a grammar that describes the very different sequence adopted in a website recipe or a recipe on the back of prepackaged frozen foods. With experience and increasing skills, grammars that are larger and more effective can be developed and applied. Much of this experience can be gained through relational concordancing as described in Section 5 below. Indeed by relating different mini-grammars in such a way that consistent patterns emerge in relation to a number of corpora, it may be possible to turn two or three mini-grammars into a single ‘not-quite-so-mini’ grammar. The second annotational tool Grammar Definition illustrated in Figure 5 is of particular relevance in this respect.



Figure 5: Grammar Definition Page: the second annotational tool

As with Systemics 1.0 (O’Halloran, Judd 2002), software for systemic functional linguistic analysis, grammars in MCA are hierarchical, consisting of sets within sets, as the rightward dislocation of subsets in Figure 5, Callout 2 and Figure 6, Callout 3 indicates. A special concordancing tool, described more fully below, the Hierarchy function (Figure 10, Callout 4, Figure 11), relates to searches through the corpus that embrace all the subordinate parameters contained within a specific set named by a ‘head’ parameter. Thus in the ‘mini-grammar’ illustrated in Figure 6, Lines is a ‘head’ parameter that allows all the subordinate parameters Line 1 to Line 20 to be retrieved. In practice, the Lines ‘mini-grammar’ allows a user to call up all the lines in a film script once the latter has been inserted into an MCA project through the third set of annotational tools incorporated in the Sequence Analysis Page.
Although this example relates to language, labels can also relate to a variety of semiotic agents and resources (e.g. music, gesture, gaze, movement and so on) in keeping with the concern with understanding the metafunctional and phasal organization of specific genres (see also Baldry 2000b, in press a; Gregory 2002; Thibault 2004). The design of mini-grammars and search mechanisms are thus motivated by the principle of choice and indeed the notion of different degrees of delicacy in choice in keeping with the systemic-functional approach to language and semiotic studies (Halliday 1978, 1989b). Some of the grammars, such as the grammar for gaze, take the form of small system networks and, thanks to the Hierarchy function, allow corpora to be searched in terms of different degrees of delicacy (Baldry, Thibault in press a). MCA grammars thus carry out different functions. In keeping with what we have stated above, they may, for example, be oriented towards foreign-language classroom concordancing and hence viewing a film script in relation to portions of a film. They may also be research oriented, designed, for example, to develop a multimodal grammar that systematically analyses the characteristics of all the film texts belonging to a particular film text genre. The car advertisement corpus (Baldry, 2004, Baldry and Thibault in press a, in press b) and the Recipe corpus are examples of this. However, a particular project can, and almost certainly will, make use of sets of labels from different ‘mini-grammars’. In other words, the overall ‘grammar’ used in a particular project and applied to a particular corpus will usually be the result of work carried out in relation to many ‘mini-grammars’ rather than one, often the work of different members of the same research team.

Figure 6: Grammar Selection Page

While the Grammar Definition function relates to the construction of the basic inventory of grammatical descriptions, another page, the Grammar Selection Page (Figure 6), permits the selection within a Project of some (or all) of the grammatical labels contained in one or more ‘mini-grammars’. This is all part of the relational philosophy adopted within MCA which is designed to promote annotation and retrieval techniques that minimise the drudgery associated with manual tagging.


Figure 7: Sequence Analysis Page: local annotation function

The final set of MCA annotation tools is found on the Sequence Analysis Page shown in Figure 7. As the callouts in Figure 7 indicate, the Sequence Analysis Page brings together both the set of sequences (Figure 7, Callout 1) annotated through the Media Indexing tools and the descriptive parameters or labels (Figure 7, Callout 4) produced through the Grammar Definition tools and made available in the specific project through the Grammar Selection tools. As a result, a much more localised level of annotation can, where necessary, be carried out. For example, for each advertisement in the TV car advertisement corpus, the actual text of a written or oral slogan has been inserted through the use of this tool in relation to the WRITTEN SLOGAN and ORAL SLOGAN parameters using the Brief description area (Figure 7, Callout 5) which, on the retrieval, concordancing side of MCA, corresponds to the Compact Search tool (Figure 10, Callout 7). When retrieved through the Search Page this allows a very clear picture of the way in which oral and written slogans function in relation to each other: one typical pattern is where a written and an oral slogan operate in complementary distribution as a pair, one relating to the car manufacturer, the other to the car model (Baldry 2004). The Sequence Analysis Page also allows users to access ancillary materials in the form of web pages, Microsoft Word documents, Acrobat Reader “*.pdf” files and various other file types (Figure 7, Callout 2).


Figure 8: Sequence Analysis Page: downloading support materials

As Figure 8 shows, these downloadable supports materials have, so far, had a primarily didactic function, providing users’ with tasks, activities, exercises and explanatory notes. Some of these relate to the static in vitro type of multimodal transcriptions and are thus designed to be used in conjunction with in vivo methods to encourage the analysis of multimodal texts from a variety of standpoints (Baldry, Thibault in press 2000b). They are linked to a particular film sequence, through the use of the Task Definition tool, shown in Figure 9, though, as in the case illustrated in Figure 8, the film may can be ‘empty’ (i.e. have with no viewable contents). The support-materials download function has been available since the second release of MCA. The availability of support materials has markedly improved the overall effectiveness of MCA; support materials containing concordancing-related examples and exercises have for example encouraged concordancing of film texts in both classroom and home settings (Baldry, Caldirola, Grunther in press; Coccetta 2003).


Figure 9: Task Definition Page: downloading support materials

4. MCA as an online multimodal concordancing system

A good online concordancer should allow users to customise the program’s functioning to suit their own particular needs. One of these needs relates to allowing researchers to carry out annotational processes offline. This requirement is not restricted to those financial and logistical circumstances where Internet access is problematic but includes, for example, those cases where annotation can be carried out more quickly and efficiently by using the word processing and spreadsheet facilities available in, for example, Microsoft Office and subsequently uploading the files containing the annotations into a specific MCA project. The functional tagging (Baldry, Thibault 2001) of scripts, with its many recurrences, would, for example, benefit greatly from such a development given the cut and paste facilities of commercial packages. Already considerable parts of the processes relating to corpus construction, annotation and document support/management are carried out offline when using MCA. For example, in the corpus construction phase, they include the conversion of the chosen texts to the *.wmv streaming video format with commercially available software. For example, recipes can be scanned and turned into a film consisting of a sequence of stills, with each still reproducing a page of recipes. Nevertheless improved upload and download tools are planned for MCA 4.


Figure 10: MCA’s Search Page

The same principle applies to the search stages of concordancing, as well as to the corpus construction and annotation stages, although in this case download rather than upload tools are involved. Figure 10 illustrates the tools incorporated in MCA’s Search Page as well as the two before and after states in which the page can exist. The before state relates to a search that has been decided but not yet executed (Figure 10, Callouts 3 to 7), the after state to the ‘results’ achieved once the search has been carried out, which are shown in the top part of the Search Page (Figure 10, Callout 2). The ‘results’ of the search take the form of rows of concordances. A third empty state is also possible, namely when no selections have been made and no results are shown, a function realised with Clear button, the last option in the bottom menu bar (Figure 10, Callout 7). This bar also contains the first embryos of the download tools that are expected to emerge with future releases of MCA. For example, the Exportable HTML function strips out the Windows Media player symbol (and of course access to a film sequence) and time-span references from the returned data leaving only the rows of annotated concordances which can be cut-and-pasted into Word and Excel documents for further processing. The Rows-to-Retrieve function (Figure 10, Callout 6), which has a default setting of 50 rows, governs the number of rows shown on any single page. In other words, when 300 rows are returned by a search using the default 50-row setting, the results will be displayed on 6 pages. If, on the other hand, the Rows-to-Retrieve function is set at 300 all the results will be housed on a single scrollable page. Clearly, the latter option is suitable when cutting-and-pasting into another software program, the former when the analysis is carried out on screen.
The purpose of the Search Page is to use search tools in combinations so that they make available sets of annotations resulting from the three stages of annotation. The form that concordances take in MCA reflects this. Concordances consist of a single compulsory line containing, inter alia, a reference to a film segment and access to it through the Windows Media Player symbol and unlimited optional subsidiary lines containing ‘mini-grammar’ labels and ‘local’ annotations (see Figures 10, 11 and 12). A search thus allows the labels and annotations associated to them to be retrieved together with the relevant portion of film. The inter-linked search tools in the Search Parameters section (Figure 10, Callout 3) allow the selection from a drop-down menu of a combination of up to three labels or descriptive parameters. These are, of course, made available in the project by the Grammar Selection tool and correspond to the labels in various mini-grammars created by the Grammar Definition tool. For example, the concordance rows relating to slogans (Figure 10, Callout 2) were created by selecting two labels: Slogan: written and Slogan: oral.
Because it is useful to find the rare cases where, at least apparently, no slogan exists in an advertisement (Baldry 2004), the annotational system adopted includes a further tag related to the Slogan: written and Slogan: oral labels which indicates their presence or absence. These tags were inserted using the Sequence Analysis annotation tool, which locally modifies a label selected from a mini-grammar. The typical result of this type of combined, relational search will be to allow a researcher to identify those texts which do not conform to an expected pattern and to view the relevant sequence of film by clicking on the Media Player symbol found in each concordance. It will then be finally possible to make a decision regarding the reasons for this exception and the actions (e.g. a further round of annotation) to be undertaken to account for the exceptions. We may note in passing that the current version of MCA requires the Sequence Analysis tag (e.g. a word or a symbol) to be typed in. Future versions of MCA are expected to provide a drop-down menu, compatible with the uploading of offline files, so that the tags can be inserted more efficiently through mouse-selection rather than through typing.
MCA was designed with the specific objective of being able to search corpora in terms of genre features, metafunctional analyses and multimodal co-patternings of semiotic resources and to divide the texts into meaningful sequences and particularly into meaning-making blocks such as phases, subphases and transitivity frames (Thibault 2000, Baldry, Thibault in press a, in press b). A significant example of this is work is given in Figure 11 where the incidence of subphases relating to driving in TV car advertisements is indicated. This is made possible by a search using the Hierarchy function (Figure 10, Callout 4) that searches through a mini-grammar relating to the material process of driving, where the head parameter is MATERIAL (169). Note that all descriptive parameters or labels in MCA’s mini-grammars are numbered, so that parameters that have been given the same name can be distinguished both by users and, of course, by MCA.


Figure 11: Search inside the hierarchy function

Lemma-based concordancing, however, is not excluded. A significant example is shown in Figure 12 where the Lines mini-grammar has been selected together with the Search inside the hierarchy function illustrated in Figure 11. A keyword ‘just’ has also been typed in. The resulting search shows the co-text of the word ‘just’ for the entire film script, as occurs with traditional lemma-based concordancing. However, the co-text is now extended from a purely linguistic co-text (and a written one at that) to a multimodal co-text. In other words, users who want to understand the varying uses of the word ‘just’ in the film in question can simply click on the Media Player symbol available in each concordance and sit back to watch, top listen and to understand. Clearly, as the work of development progresses, efforts are being made to integrate both approaches at the level of the design of MCA itself. In this respect a major step in implementing online concordancing was the incorporation from the second release of MCA onwards of a Task Definition tool that allows documents containing course materials, either those for a whole course or for particular units, to be retrieved through the Search Page. MCA 4, now in the design phase, vastly improves the integration between course materials and corpus.

















Figure 12: Search Page: downloading support materials

5. Security management

Why is it that MCA exists as an online tool rather than as a CD-ROM? There are many answers to this question. Cost-effectiveness is certainly a major consideration. MCA is an interface to Microsoft Sequel Server which, in turn, requires the use of Microsoft 2000 Server. If distributed by CD-ROM, as well as having to buy licences for these relatively expensive software tools, users would face the lengthy and often costly installation, management and maintenance procedures associated with running a server as opposed to a client computer. Apart from guaranteeing free and immediate access to the current version of MCA, an online approach also allows frequent upgrading to be made without distribution costs. One potential disadvantage, however, relates to the question of privacy. While an online system undoubtedly brings with it the benefit that the concordancing of a particular corpus can potentially be carried out by a very large number of users, more often than not, it is necessary to restrict user access. One example is the University of Pavia’s short courses in foreign languages whose materials are delivered through MCA, with access restricted, mainly for financial and copyright reasons, to students enrolled in the University of Pavia actually attending these courses. Another is the case of a group of researchers within the COFIN projects mentioned above who, in the development phases, need to restrict access to members of the group actively working on a specific corpus.
The solution to this problem is a security system based on the familiar combination of user names and passwords that functions to restrict user access to projects and tools. Each Project or Grammar (or both), or indeed groups of them, can be ‘protected’ in such a way that different users will have, for example, the following access rights: (a) no access; (b) partial access e.g. non authorial ‘browsing’ rights; (c) full access allowing contents and functioning to be changed; (d) local administrator access that not only allows changes in contents and functionality but also allows local administrators to create their own profiles granting the right (or otherwise) for ‘lower level’ local administrators to modify a project or a mini-grammar.
Many different profiles can, of course, be linked to each Project or Grammar. A user’s level of authorisation and access to projects and grammars is defined when a user logs on. Unregistered users, those without a user name and password, can access the About MCA function on MCA’s Home Page (see Figure 1, Callout 2) which gives a basic illustration of the way MCA works and the background to its development. It is also possible to become a registered user through the New Registration tool. Authorisations are granted by local administrators participating in the project who identify users on the basis of the email used and other registration data. For example, in the short courses run by the University of Pavia, students who register with MCA for a particular course are required to give the code name for the course as a means of identification. Where users are not authorised to access a particular function, MCA will behave in one of two ways: in some cases the function will simply not be visible on screen, whereas in other cases an “MCA exception” message will appear indicating that access has not been authorised for the particular user.
The basic mechanism used in MCA to implement this security system is the profile as indicated in Figure 13. A profile is essentially a linkage between a list of users (User Group) who are authorised to access a set of MCA functions (called Operations or Classes) and a particular Project or Grammar. The Security page is used to create profiles and the set of links they involve, for example, to associate (Figure 13, Callout 3) or revoke (Figure 13, Callout 7) a particular function. The example given relates to profiling but a similar associate/revoke principle exists for the other tools accessible through the ToolBox in the Security Page (Figure 13, Callout 1). The articulation into Users, User Groups, Operations and Classes allows a modular approach to be adopted. For example, User Groups group together individual Users, each with predefined characteristics but a local administrator can associate/revoke any User’s access rights with a single mouse click.


Figure 13: Security Management Page showing editing possibilities

6. Conclusion

The paper has reconstructed MCA’s development path from its origins in 1998 to the current position (October 2004). MCA is part of research which attempts to provide a new chapter in multimodal research concerned ultimately with a corpus-based approach to multimodal text analysis and transcription (Baldry, Thibault in press a, in press b) and with multimodal corpus linguistics (Baldry 2000a, 2000b). MCA allows research to be carried out into film corpora and mixed media corpora within a variety of theoretical frameworks, the most prominent of which has, so far, been a broadly systemic functional approach to the analysis of film texts. To judge from the seminal work carried out in the mid-1990’s into multimodality (Kress and van Leeuwen 1996, O’Toole 1994), with very few exceptions (Baldry in press b, Baldry, Prozzo 1996, Martinec 1996, 1998, Thibault 1990) little multimodal film text analysis appears to have been carried out in this period and certainly none within a corpus-based strategy. As explained above, film texts are typically taken to include texts which are only partly film-like such as websites or those, such as illustrated recipes in cookbooks, which are picture-like.
From the outset, MCA has been designed as a flexible system that can evolve gradually, growing and changing, as needs change and as new panoramas open up when one height has been scaled. The development is by no means complete but what has been achieved so far has, as predicted, (Baldry, 2000: 36-37) allowed a new branch of corpus linguistics, namely multimodal corpus linguistics to emerge (Baldry, Thibault 2001; Baldry, Thibault in press a, in press b). Some indication of the way in which MCA contributes to this nascent field and its classroom applications is given in the various publications quoted in this paper.
Much of the development of MCA relates to the positing of user scenarios. A basic question since the outset has been: “In what ways can MCA be modified to create new and better services for users?” This approach has meant that the number of MCA users has grown steadily from a minuscule number to several thousand per year. Much of this increase is a reflection of the classroom appeal of concordancing techniques of multimodal corpora that combine film scripts with film texts and which hence extend ‘traditional’ keyword concordancing (Sinclair 1991, Partington 1998) from language-only texts and their co-texts to multimodal texts and multimodal co-texts (Baldry, Caldirola, Grunther in press). However, although some of the original goals have changed and with them some of the design features, the thinking behind MCA remains the same, the development of a system that makes it easy to study mixed-media corpora (as defined above) in which film text can be studied in terms of in vivo and in vitro formats
As a research tool, MCA allows researchers to characterise the meaning-making processes in film texts more effectively and efficiently. Researchers who posit the existence of various types of functional units in film texts or the existence of typical resource co-deployments used to create each of these units, can test out their hypotheses in relation to a substantial number of texts. All of this leads to a further round of formulation of hypotheses about the characteristics that texts share, the segmentation and annotation of texts and their parts, the concordancing of annotated texts and analysis of the data in the concordances thus produced. MCA has thus been designed to promote research into the nature, characteristics and functions of texts and genres. By encouraging the research procedure described above, MCA leads to the identification of what is recurrent in the meaning-making processes of film texts and to the establishment of ways of organising these recurrences into types (Baldry 2004, Baldry, Thibault 2001, in press a, in press b). By searching annotated corpora, patterns relating to metafunctions (Halliday 1978, 1994), phases (Gregory 1995, 2002) and generic structure (Hasan 1984) can be brought out. In particular, MCA promotes the identification and reconstruction of recurrent part-whole relationships in film texts and genres (Baldry, Thibault in press b; see also Cheong 2004 who looks at generic structure in multimodal print advertisements).
As a corpus construction tool, MCA enables authorised users to segment film into functional units and, while viewing them, type out detailed annotations relating to patterns of meaning made by the various semiotic resources deployed within each unit. As explained above, these annotations are based on selections from sets of descriptive labels or ‘mini-grammars’ created using MCA’s annotational tools. As a search tool, MCA provides concordances of film texts that bring out recurrent patterns in these texts within the overall goal of establishing a multimodal grammar and a multimodal syllabus relevant to the needs of scholars and students studying, for example, English-language film texts (Baldry 2000a, 2000b, Thibault 2000).
Finally, the paper has suggested, in outline, the development path that MCA is expected to take in the near future in the light of the constructive criticisms that have emerged from users. Naturally, one of the main problems is to decide which design features to prioritise given the small software development team and the limited budget available. Even so MCA is fulfilling a role not yet contemplated by commercial software.

7. Acknowledgements
MCA has been built as part of three inter-university projects which have received substantial funding from the Universities involved (Padua, Pavia, Pisa and Trieste) and from the Italian Ministry for Universities and Research (MIUR formerly MURST). The forerunners of MCA were built as part of the Citatal Project (1998-2000), while MCA 1 and (most of) MCA 2 were built as part of the Linguatel Project (2000-2002). Both of these projects were co-ordinated by Carol Taylor Torsello (University of Padua). Some of MCA 2 and all of MCA 3 have been developed as part of the Didactas Project (2002-2004) co-ordinated by Christopher Taylor (University of Trieste). It is expected that the next release (MCA 4) will be online early in 2005.; Sheila McVeigh produced the PETSIM corpus/project featured in Figure 8. Patti Grunther developed the CBI corpus/project featured in Figure 12.

8. References

Aston, Guy (1997) “Small and large corpora in language learning". In B. Lewandowska-Tomaszczyk & P.J. Melia (eds), PALC 97: Practical applications in language corpora. Lodz: Lodz University Press. 51-62.
Aston Guy, Burnard, Lou (eds.), (2001) Corpora in the description and teaching of English, Bologna, CLUEB.
Baldry, Anthony (2000a) “Introduction”, in Anthony Baldry (ed.), Multimodality and Multimediality in the distance learning age, Campobasso, Palladino Editore, pp. 11-39.
Baldry, Anthony (2000b) “English in a Visual Society: comparative and historical dimensions in multimodality and multimediality”, in Anthony Baldry (ed.), Multimodality and Multimediality in the distance learning age, Campobasso, Palladino Editore, pp. 41-90.
Baldry, Anthony (2002a) “Computerised subtitling: a multimodal approach to the learning of minority languages”, in George Talbot and Pam Williams (eds.) Essays in Language, Translation, and the Digital Learning Technologies in honour of Doug Thompson Coventry, England, Troubador, 69-74.
Baldry, Anthony (2002b)‘Multimodal concordancing and screen translation’, RILA (Rassegna Italiana di Linguistica Applicata), XXXIV,1-2: Rome, Bulzoni, 2002, pp. 161-175.
Baldry, Anthony (2003) “From multimodal transcription to multimodal concordancing: an analysis of TV adverts” in Flora Sisti (a cura di) Insegnamento delle lingue nei nuovi corsi di laurea. Trieste, Edizioni Goliardiche, pp. 61-87.
Baldry, Anthony (2004) “Phase and transition, type and instance: patterns in media texts as seen through a multimodal concordancer” in Kay O’Halloran (ed.), Multimodal Discourse Analysis: Systemic Functional Perspectives, London and New York, Continuum, pp. 83-108.
Baldry, Anthony (in press a) “Concordancing film corpora: the role of MCA in multimodal discourse analysis” in Carol Taylor, Maria Grazia Busà, Sara Gesuato (eds.), Proceedings of the Seminar, Linguatel: Progetto telematico di lingua e mediazione interlinguistica, 9-10 May 2002 – University of Padua, Unipress, Padova.
Baldry, Anthony (in press b) “Children’s use and awareness of genre: A case study of the evolution from multimodal transcription to multimodal concordancing based on system networks” in Carol Taylor Torsello, Maurizio Gotti and Christopher Taylor (eds.) I Centri Linguistici: approcci, progetti e strumenti per l’apprendimento e la valutazione. Atti del 3° convegno nazionale AICLU, Trieste: EUT.
Baldry, Anthony (in press c) ‘The role of multimodal concordancers in multimodal corpus linguistics’ in Terry Royce and Wendy Bowcher (eds.) New directions in the analysis of multimodal discourse, New Jersey, Lawrence Erlbaum Associates.
Baldry Anthony, Caldirola Elena, Grunther Patti (in press). “Knowing and learning about what to expect: the role of multimodal concordancing in explorations of film texts in foreign language short courses” in Proceedings of the Sixth Aiclu Seminar Letojanni, Messina 30-31 October 2003.
Baldry, Anthony, Prozzo, Nicola (1996) Ipermedia nell’educazione linguistica, Roma, Armando.

Baldry, Anthony, Taylor, Christopher (2002) “Multimodal Corpus Authoring System: Multimodal Corpora, Subtitling and Phasal Analysis” in Proceedings of the Multimodal Workshop, LREC Congress, Las Palmas, June 2002.
Baldry Anthony, Taylor, Christopher (2004) “Multimodal concordancing and subtitles with MCA” in Alan Partington, John Morley and Louann Haarman (eds.) Corpora and Discourse, Bern, Peter Lang, pp. 57-70.
Baldry, Anthony; Thibault, Paul (2001) “Towards multimodal corpora” in Guy Aston, Lou Burnard (eds.), Corpora in the description and teaching of English, Bologna, CLUEB, pp. 87-102.
Baldry, Anthony; Thibault, Paul (in press a) “Multimodal corpus linguistics” in Geoff Thompson and Susan Hunston (eds.), System and Corpus: Exploring Connections, London, Equinox.
Baldry, Anthony; Thibault, Paul (in press b) Multimodal Transcription and Text Analysis, London and New York, Equinox.
Biber, Douglas, Conrad, Susan, Reppen Randi (1998) Corpus Linguistics: Investigating language structure and use. Cambridge, Cambridge University Press.
Cheong, Yin Yuen (2004) “The construal of Ideational meaning in print advertisements” in Kay O’Halloran (ed.), Multimodal Discourse Analysis: Systemic Functional Perspectives, London and New York, Continuum, pp.163- xxx.
Coccetta, Francesca (2003) Language functions in films: a multimodal corpus for English language teaching. Unpublished degree thesis. Dipartimento di Lingue e Letterature Anglo-germaniche e Slave, University of Padua.
Gregory, Michael (1995) “Generic expectancies and discoursal surprises: John Donne’s The Good Morrow”, in Peter H. Fries and Michael Gregory (eds.), Discourse in Society: Systemic Functional Perspectives. Meaning and choice in language: studies for Michael Halliday, 67-84, Norwood, NJ., Ablex, pp. 67-84.
Gregory, Michael (2002) “Phasal analysis within communication linguistics: two contrastive discourses” in Peter Fries, Michael Cummings, David Lockwood and William Sprueill (eds.), Relations and Functions in Language and Discourse. London and New York, Continuum, pp.316-345
Hasan, Ruqaiya (1984) “The structure of the nursery tale: an essay in text typology” in Lorenzo Coveri (ed.), Linguistica Testuale: Proceedings of the 15th International Congress of the Italian Linguistics Society, Roma: Bulzoni, pp. 95-114.
Hasan, Ruqaiya (1996) Ways of Saying: Ways of Meaning, Cassell, London.

Halliday, M.A.K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
Halliday, M.A.K. (1989a [1985]). Spoken and written language. Oxford: Oxford University Press.
Halliday, M.A.K. (1989b). “Part A.” In Halliday, M.A.K., Hasan, Ruqaiya, Language, context, and text: aspects of language in a social-semiotic perspective. Oxford: Oxford University Press, pp. 1-50.
Halliday, M. A. K. (1994 [1985]). An Introduction to Functional Grammar. 2nd edition. London and Melbourne: Arnold.
Iedema, Rick (2001) “Analysing Film and Television: A Social Semiotic Account of ‘Hospital: An Unhealthy Business’” in Theo van Leeuwen, Carey Jewitt (eds.) The Handbook of Visual Analysis, London: Sage, pp. 183-204.
Kress, Gunther; van Leeuwen, Theo (1996) Reading Images. The Grammar of Visual Design. London and New York, Routledge.
Lemke, Jay 1998 “Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text” in Jim Martin and Robert Veel, eds., Reading Science. London: Routledge, pp. 87-113.
Lidgley, Deborah (1998) Analisi di testi multimodali di bambini inglesi. Unpublished graduation thesis. Dept. of Foreign Languages and Literature, University of Pavia.
Lidgley, Deborah (2000) “Distance learning through multimedia techniques: a self-access seminar on children and media” in Anthony Baldry (ed.), Multimodality and Multimediality in the distance learning age, Campobasso, Palladino Editore, pp. 179-184
Lombardo, Linda (2001) Selling it and Telling it. A functional approach to the discourse of print ads and Tv news. Roma, Istituto Linguistica Moderna, Luiss, Guido Carli.
Martinec, Radan (1996). “Rhythm in multimodal texts”. Paper presented at the First Action Colloquium, June 1996, London College of Printing, London Institute: Mimeo.
Martinec, Radan (1998) “Cohesion in action”, Semiotica 120-1/2: pp. 161-80.

O’Halloran, Kay. (2003) “Systemics 1.0: Software for research and teaching systemic functional linguistics” in RELC: A Journal of Language Teaching and Research in Southeast Asia, 34.2 (2003), 155-177.
O’Halloran, Kay (2004) “Visual semiosis in film” in Kay O’Halloran (ed.), Multimodal Discourse Analysis: Systemic Functional Perspectives, London and New York, Continuum, pp. 109-130.
O’Halloran, Kay (2005) Mathematical Discourse: Language, Symbolism and Visual Differences, London and New York, Continuum.

O’Halloran, Kay, Judd, Kevin (2002) Systemics 1.0 CD-ROM, Singapore: National University of Singapore.
O’Toole, Michael (1994) The Language of Displayed Art, London: Leicester University Press.
Palumbo, Roberta (2000) Experimenting with narrative genre: children’s storytelling in a multimodal computer environment. Unpublished graduation thesis. Facoltà di Lingue e Letterature straniere, University of Venice.
Partington, Alan (1998). Patterns and Meaning. Amsterdam: Benjamins.
Sinclair, John (1991). Corpus, concordance, collocation. Oxford: Oxford University Press.
Taylor, Christopher (1999) The Translation of Film Dialogue, in Susan Bassnett, Rosa Maria Bollettieri Bosinelli, Margherita Ulrych (eds.) Textus, Vol. XII, No. 2, eds. Tilgher: Genova.
Taylor, Christopher (2003) “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films”. In The Translator 2003 9 (2)191-205.
Taylor, Christopher, Baldry Anthony (2001a) “Computer assisted analysis and translation: a functional approach in the analysis and translation of advertising texts”, in Erich Steiner and Colin Yallop (eds.), Exploring Translation and Multilingual Production: Beyond Content. Berlin, Mouton de Gruyter, pp. 277-305.
Taylor, Christopher, Baldry Anthony (2001b) “Computer-assisted analysis and translation (characteristics of interactive self-access computer modules incorporating a functional approach in the analysis and translation of advertising texts)”, in Carol Taylor Torsello, Giuseppe Brunetti, and Nicoletta Penello (eds.), Corpora testuali per ricerca, traduzione e apprendimento linguistico. Studi Linguistici Applicati. Padova, Unipress, pp. 273-292.
Taylor Torsello, Carol, Giuseppe Brunetti and Nicoletta Penello (eds.). 2001. Corpora testuali per ricerca, traduzione e apprendimento linguistico, Padova, Unipress.
Taylor Torsello, Carol, Maria Grazia Busà, Sara Gesuato (eds.). In press. Lingua inglese e mediazione linguistica. Ricerca e didattica con supporto telematico, Padova, Unipress.
Thibault, Paul J., (1990) “Questions of genre and intertextuality in some Australian television advertisements”. In Rema Rossini Favretti (ed.) The Televised Text, Bologna: Patron, pp. 89-131.
Thibault, Paul (2000) “The multimodal transcription of a television advertisement: theory and practice”, in Anthony Baldry (ed.), Multimodality and Multimediality in the Distance Learning Age. Campobasso, Palladino Editore, pp. 311-385.
Vasta, Nicoletta (2001) Rallying Voters: New Labour’s Verbal-Visual Strategies. Padova, Cedam.
Ventola, Eija, Charles, Cassily, Kaltenbacher, Martin (eds.) In press. Perspectives on Multimodality, Amsterdam: John Benjamins.
Zago, Silvia (2002) A multimodal analysis of six television adverts. Unpublished graduation thesis. Dipartimento di Lingue e Letterature Anglo-Germaniche e Slave, University of Padua.