I’m slowly working my way through Kress and van Leeuwen’s Multimodal Discourse: The Modes and Media of Contemporary Communication. While Reading Images was concerned with the printed page and the mixture of text, images and ornamentation, In Multimodal Discourse, the authors have tried to broaden the theory to account for all modes of communication based on “analysis of the specificities and common traits of semiotic modes which takes account of their social, cultural and historical production, of when and how the modes of production are specialized or multi-skilled, hierarchical or team-based, of when and how technologies are specialized or multi-purpose, and so on.” (p. 4)
In other words they want a really big theory that accounts for anything. In the process of describing this theory they’ve confused me at almost every turn. Some of the theorizing they admit is inconsistent and incomplete (see the Preface). Is color a mode? Is the type and thickness of paper a mode? Gestures? Tone of voice? Architectural features of buildings? The reader is right in asking, “What isn’t a mode then?” Of course this type of theorizing is just a way of proving the center by starting from the boundaries. Whereas traditionally discourse has been conceptualized as speech reduced to typewritten form and then analyzed in splendid isolation, social semiotics shows how many things go into communication and how vested it is in human interaction.
As a way of getting my head around the basics of Kress and van Leeuwen’s book I’m going to look at “Directions,” that is, how do we get information about how to arrive at an unknown destination. I’ll do that by looking at some of my recent travels (or should I say travails) around Cape Town.
Home Affairs
At the beginning of the year, I needed to go to Home Affairs and begin the process of applying for a study permit. I had visited Home Affairs before and so it was easy to locate with a little help from my wife who was giving me directions from the passenger’s seat. But this time when I arrived, I discovered that Home Affairs had moved. There was a sign on the door saying “Home Affairs is now located at 56 Barrack Street.” After studying the sign for a moment, I approached someone sitting behind a reception desk and asked where 56 Barrack Street was. He helpfully left his desk, came out of the building with me and began to describe in heavily accented English and much gesturing how to get to 56 Barrack Street. After he was done, I thanked him, took one more look at the sign and then got into my car and looked up 56 Barrack Street on my smart phone. Google Maps located it (although it was mislabeled) and I was even able to turn Navigation on and have my phone talk me through the process of getting to 56 Barrack Street.

Home Affairs in Cape Town labeled Faircape Management by Google Maps
Now, in the preceding paragraph you can see a variety of modes that were used to get directions. First, there was my unspoken recollection (mode: memory) of where I thought Home Affairs was located. That was accompanied by my wife giving me instructions (mode: speech) from the passenger seat. Then I saw the sign (mode: printed announcement) which was followed by someone giving me directions (modes: speech and gestures). After that we looked up the directions on Google Maps (mode: Internet web resource) and then made use of navigation: (mode: visual and voice GPS).
Each of these modes has its own advantages and disadvantages. The helpful man knew where it was and was able to give me very specific directions. But I couldn’t understand his speech very well and there were so many steps that I didn’t even try to remember them all. Instead I remembered only that physical address “56 Barrack Street” and turned to Google Maps to help me. Google Maps is very powerful but as you can see in the image it mislabeled the place I was going to and the map itself had far too many details to absorb (especially while driving). GPS Navigation had the advantage of locating where my car was and planning out a route that would put me near 56 Barrack Street. But it wasn’t aware of traffic congestion and route changes due to road construction.
Multimodal Discourse analysis would say that Directions is a discourse and that it is able to be expressed in a variety of modes. Which modes I was able to tap into would determine how I went about getting to Home Affairs and the relative ease of doing so. A month later, I was on the train heading for Home Affairs again. This time I could call on my previous knowledge. I knew Home Affairs had moved to 56 Barrack Street and that this was located in walking distance of the Cape Town train station. I verified the location on Google Maps. I noticed that as I walked to Home Affairs I would have to pass Longmarket, Caledon and Albertus streets. The next street is Barrack Street. So I confidently headed toward my destination, passed Longmarket, but then I ran into trouble. I didn’t see a sign for Caledon, or Albertus, or Barrack. I walked as far as the Parliament and realized I was lost (mode: visual). I looked up my destination again on Google Maps and discovered I had overshot Barrack Street. Turning around I headed back two blocks, saw a sign for Barrack Street and carried on to my destination.
In this second trip we can see that another mode, street signs, didn’t work out for me very well. Partly this is because not every street had a sign at the intersection. But also, the Google Map I had referenced gave me less detail than I needed. I didn’t realize that Longmarket, Caledon and Albertus are just three of many streets in that area and in fact Albertus didn’t connect with the street I was on.
So where are we? Multimedia discourse has given us a way of talking about a large number of disparate activities as actually belonging to a single discourse: directions. By considering each mode in turn we’ve been able to see the strengths and weaknesses of each mode and also see how they are often used together to guide human behavior.
I called on a rich set of modes in order to achieve my goal of getting to 56 Barrack Street. A person with only a street address would have to rely on other modes such as maps, asking for directions on the street, and probably a lot of wandering around.
What happened once I got inside the door is another story. I had to find out where to go, which line to stand in, how to participate in the choreography of seating. But that’s a discourse for another day!
Follow-up to this post: Directions, again. Design as mode.