[ IIMS 94 contents ]
MultiMak: A multimedia system for the teaching of sign language
George Lee Stuart
Southern Cross University, Lismore NSW
The teaching of sign language to hearing persons is currently achieved by means of formal lectures, seminars, picture books and video tapes. Each of these delivery methods has inherent disadvantages either in terms of cost, logistics and/or the quality of the delivery medium. Teaching sign language to hearing persons paradoxically requires visual and auditory stimulus, together with a feedback mechanism for the student. This paper describes the design and development of a multimedia system, MultiMak, for reaching the Makaton Vocabulary sign language to hearing persons. The system uses text, drawings, sound and interactive video, implemented via QuickTime, to present a dynamic and truly interactive teaching system.
Background
The MultiMak system evolved over the past three years in response to a perceived need in terms of an effective delivery system for the teaching of the skill based activity of language signing and the gradual availability of the enabling technologies.
I was presented, in April 1990, with the task of learning sign language in the Makaton Vocabulary. The delivery mechanism for the vocabulary was attendance at sign workshops and a manual of line drawings (Cooney, Cowley & Knox, 1984). There were a number of drawbacks in this mechanism, namely the workshops were infrequent and casual contact with signing instructors was irregular and did not provide an effectively systematic learning program. I should state that this was a result of my own work schedule and not due to any lack of commitment and effort on behalf of the signing professionals.
The line drawing manual was designed as an aide memoir for students and tutors and proved to be of only partial use in the learning of the vocabulary by self study. Some signs, such as BROTHER, are easily interpretable from their line drawing representation and accompanying narrative, Figure 1(a) whilst others, such as HERE, are less so, Figure 1(b).
Figure 1
Although all of the line drawings are accompanied by narrative descriptions of the appropriate sign motion, even with this description, the process required is not always clear. For a full understanding of the correct motion one really has to see the sign performed. There is a video tape of the signs available through the signing agencies but this was not available to me in April 1990.
The delivery system then, was ineffective, inefficient and untimely for a large number of persons who might otherwise learn the vocabulary. This was in no part due to the formidable efforts of the signing tutors. Rather the problems of teaching sign language are inherent within the medium. It was clear that a computerised system could be of some assistance, but what kind of system?
The first phase of a computerised signing system
HyperCard became available to me in 1991 and a demonstration of its animation capabilities invoked the genesis of the first signing system. This system formed the basis of a final year project for three Bachelor of Business students. Their brief was to produce a signing system, using the Makaton vocabulary, in HyperCard which would animate the line drawings of the vocabulary manual. In addition sound was to be incorporated to provide information to the none reading user such as a young child. The result of this project was a partially completed system called Hypersign (Friend, Sallaway & Tucker, 1992).
Hypersign
Hypersign provides four functional components to the user. These are a tutorial, a browser, a dictionary and a self test. These components are listed as a selectable menu under the heading of OPTIONS in the opening card of the Hypersign stack, Figure 2.
Figure 2
When selected, each of the four components allows the user to choose one of the stages of the Makaton Vocabulary. For example selecting the tutorial option produces Figure 3.
Figure 3
This second card is in fact the opening card with a number of text fields removed and others substituted. Makaton has nine stages of sign, represented by the buttons of the second screen of a particular component, ranging from the conceptually and implementationally easy such as BROTHER from stage one, Figure 1(a), to more difficult signs such as HERE, Figure 1(b). In order to provide the basis for sign illustration each of the line drawings from stage one was digitally scanned from the Makaton manual (Cooney, Cowley & Knox, 1984) and incorporated into a separately named sign card. This is illustrated by the left hand side of Figure 4, representing the BROTHER sign.
Figure 4
The Hypersign sign cards
Some of the signs were perceived to be readily animated, such as BROTHER, Figure 1(a) and Figure 4. This sign was animated by simply swapping between two cards with the hands reorientated so that the left becomes higher than the right and then a return to the original card appears to place the right hand above the left thereby giving the illusion of motion. Other animated cards use similar visual tricks to achieve the illusion of motion. Yet others signs were deemed to be much more difficult to animate such as DRINK, Figure 5. This sign requires a twisting of the wrist as well as an upward motion to the mouth with the thumb.
Figure 5
The Hypersign card buttons
Each of the sign cards share a homogenous background containing buttons which sometimes appear or which are sometimes hidden depending on the function of the given card. The home card, Figure 2, has no navigation buttons only the Quit and HELP buttons. The Brother card, Figure 4, in contrast has all of its six possible buttons visible. The HOME button returns to the opening card, LAST goes to the previous sign and NEXT to the next sign. The RETURN key sends the user to the appropriate first card for the current function. In our examples this would be a return to Figure 3, the Tutorial card. The BROTHER card, Figure 4, also illustrates the PLAY button. This button appears whenever a sign a animated and may be repeatedly pressed without leaving the sign.
The dictionary
If the user wore. to select the dictionary function then they would see the Dictionary card, Figure. 6. This card is the same card as the tutorial card but with different text fields revealed. The dictionary component allows the user to select the stage as before and a scrolling menu is presented in the left hand window, Figure 7. The dictionary component visits the sign card of choice, only once, and then returns to the Dictionary card, Figure 7 when the return button is pressed.
Figure 6
Figure 7
The browser
The browser was essentially a repeat of the tutorial stage.
The self test
The self test was not fully implemented.
Sound in Hypersign
Each of the sign cards implemented in Hypersign has the sign spoken to the user. This is achieved by the insertion of a sound using the standard HyperCard audio input facility and a standard Apple microphone, Figure 8.
Figure 8
This utility places the sound in a HyperCard button which is then made transparent and invoked from the card script with the play command, Figure 9(a) and 9(b).
 |
 |
| (a) | (b) |
Figure 9
Hypersign outcomes
Hypersign was only a partial success. Signing professionals did not like the animations as they did not accurately mimic the sign as performed by a human. There were some reservations about the user interface too, especially in regard to young children and those who were computer phobic.
The arrival of QuickTime at the university in 1992 offered a solution to the first of these objections and a redesigned user interface lead to the development of MultiMak.
MultiMak
MultiMak has two principal differences from Hypersign. Firstly there is a redesigned user interface and secondly MultiMak incorporates QuickTime movies of the signs being presented.
The MultiMak user interface
A conscious decision was taken to attempt to orientate the new MultiMak system towards children. The reason for this decision is that it is hoped that eventually MultiMak may become a common resource in schools. To this end an attempt was made to keep the usage of the system simple and also to use written language that did not frighten off young children. Also buttons were designed to be sufficiently large to accept touch screen input and there is no requirement for keyboard input. This could be useful to the many special needs children who need to learn sign language. Just pressing a button and seeing and hearing the sign can be of enormous benefit to such children many of whom appear to exhibit increased attention span when using a computer.
The MultiMak introduction
The way in which the user is introduced to the system has been changed. It was felt that a more gentle introduction, with more opportunity for gleaning information about MultiMak was required. A number of pages quickly introduce the user to the system. A time delay and then fade to next page was introduced for Figures 10(a). A click right arrow procedure then presents pages 10(b), 10(c) and 10(d). A brief audio message welcomes the user.
The user selects the activity from the given choice, Figure 10(d). Each of the part buttons highlights whenever the cursor enters them in order to emphasise their special function. This screen, Figure 10(d), is called the "part selection page".
(a)
(b)
(c)
(d)
Figure 10
The tutorial system
The tutorial system navigation is essentially the same as for Hypersign and still uses forward and backward navigation. Buttons have been redesigned and relocated in order to place buttons with a similar function next to each other. Hypersign had unnecessary complications for the first and last signs of a stage by removing the NEXT and LAST buttons. MultiMak simply cycles around to the first and last sign as appropriate. The background has been redesigned on an asthenic basis. The left hand window now includes a QuickTime movie of the sign being presented. When the user enters the tutorial system they have the option of soliciting help about the system, Figure 11.
Figure 11
The help option contains information regarding the positioning of pictures and also the function of the MultiMak control buttons. The user is then presented with the "Stage Menu Page, Figure 12.
The help option contains information regarding the positioning of pictures and also the function of the MultiMak control buttons. The user is then presented with the "Stage Menu Page, Figure 12.
Figure 12
The user selecting the help option from Figure 11 has the various buttons and white spaces explained before reaching this page, Figure 12. Selecting a stage button places the user in a sign page, Figure 13.
Figure 13
The browser
The browser is radically changed and uses the static frame technique as recommended in the HyperCard Stack Design handbook (Apple Computer Corp, 1989). This new browser allows the user to access any stage and sign from any other card by selecting the stage from a pull down menu and thence the sign of choice in that menu, Figure 14.
Figure 14
The sign is displayed and the browser returns the user to Figure 14 when the return button is clicked. The browser interface is currently under review as the scrolling menu is not amenable to use with touch screens.
The dictionary
lie dictionary has been changed to present a book metaphor. All of the signs are included in alphabetical order on the left hand page. They are presented as transparent buttons. The user selects a sign by clicking on the word for that sign. When a sign is selected the appropriate QuickTime movie is played on the right hand page (Figure 15). The REPLAY button is provided to enable the user to play the sign of choice as many times as they like.
Figure 15
Implementation issues
Movie acquisition and reproduction
The movies in MultiMak were filmed using standard VHS tape using an inexpensive commercially available video camera. There was no studio available for the filming and so an ordinary room was used with no special lighting. This was considered acceptable as the project was one to prove the feasibility of the technology rather than specifically to produce a polished product.
The video clips were digitised with a Video Spigot card using the Screenplay 1.02 software supplied with the card. Frame capture rates were 10 to 20 frames per second and the resultant clips were saved as Video images with a quality setting of "High". The clip sizes were typically of the order of 400 to 800 kB.
The video clips are called from within the script for each sign card by calling the QuickTime XCMD command Playmovie.
File size
MultiMak is big. The first three stages have occupied 39 Mega Bytes implying a potential total size of 120 Megabytes for the full system. This requires an ideal distribution media which is not available to us at present. Either we could distribute the system on 120 magnetic discs or we could master one CD-ROM at a cost of approximately $8,000 (at November 1993 prices), a cost burden which would have to be passed onto users. Also whilst CD-ROMs are now becoming more popular, they are by no means ubiquitous. A similar argument holds for such media as Zytech discs.
One potential solution to the distribution problem may be the data storage version of the Sony MicroDisc (MD) system when it becomes available. Such a device would be able to span the divide between computer peripheral and consumer product with all of the economies of scale and increased distribution possibilities that that implies.
The test system
The test system has not as yet been implemented. The system will perforce be a trust based one as there is no feasible way that the system can assess the users signing ability with current technology. The test system will incorporate two approaches to self testing. The user will be able to request that a randomly selected sign is presented together with a menu of possible answers. This precludes the need for keyboard input. The second approach is, again, for a randomly selected word to be provided for the user to try and sign and then the system will show the correct signing for that word so that the user may assess their skill. It is envisaged that the test system will provide signs on either a completely random basis or randomly within a particular stage by clicking the TEST button in any particular card.
The audit system
It is intended that each card and button in the system be placed in an audit trail by writing its selection and a time stamp to a log file. Post usage auditing will then be possible. This will be useful in providing a user profile in terms of the difficulty in learning Makaton, useful time frames for effective learning and also in experimentation regarding button and window placement.
It is intended to incorporate an optional questionnaire into the final system to solicit users views on the value of the system.
MultiMak's current position
The current version of MultiMak constitutes a second prototype of the system. There are a number of programming instances in the underlying Hyperscript which exhibit somewhat less than the best programming practice. In addition the test and audit systems need to be developed. The look and feel of the program need to be improved and certain operations need to be speeded up.
MultiMak is now clearly ready to enter a new phase of development. The technology has been proven to work in the context of the project. Storage and issues relating to the speed of target machines will resolve themselves over the next phase of design and development by the simple advance of the hardware norms in the installed computer base.
MultiMak, the future
A complete re engineering is now required to bring this prototype forward into a genuine product meeting market standards. This will require money and resources in excess of those currently available and the producers of MultiMak will have to expend their best efforts in that direction in the near future.
With the recent arrival of QuickTime for PCs and the ScriptX cross platform scripting language from Kaleida labs (Kaleida Labs, Inc. 1993), there is the potential to port MultiMak to PC based platforms. Indeed if the potential of ScriptX matches up to that anticipated by the advance information available, we may be able to provide MultiMak on consumer based multimedia players. Specific areas of MultiMak that need further development include:
A complete rewrite of the code possibly using ScriptX
- A re engineered interface with a professional "look and feel" including colour and buttons that look like buttons
- A new browse section to facilitate touch screen selection
- A self test system
- Professional videos and sound recording using a proficient signer (preferably a young person)
- Foreign language versions
Conclusion
The application which MultiMak addresses is the quintessential multimedia application. A system which can present a predominantly visual and/or auditory based skill to a disperse set of users who gain the added ability to interact with the system. The user is empowered, they can use the system when they want, as many times as the want, for as long as they want. They can set their own goals or be guided by a predetermined study schedule. They can use the system as a reference in terms of a signing dictionary and they can even enjoy a browse through the signs to gauge the waters.
Bibliography
Apple Computer Corp (1989). Hypercard stack design handbook. Addison Wesley, Reading Mass.
Cooney, A., Cowley, J. & Knox, G. ( 1984 ). Line drawings for use with the revised Makaton Vocabulary (Australian Version). Makaton Australia.
Friend, D., Sallaway, T. & Tucker, S. (1992). Hypersign manual and report. BBus. Project, University of New England - Northern Rivers.
Kaleida Labs, Inc. (1993). Kaleida, ScriptX and the multimedia marketplace. Kaleida labs, Inc.
Author: Mr George Lee Stuart, Lecturer in Computing
Southern Cross University, Lismore NSW 2480, Australia
Tel. + 61 (066) 20 3704 Fax. +61 (066) 22 1724
Internet. GStuart@alsvid.une.edu.au
Please cite as: Stuart, G. L. (1994). MultiMak: A multimedia system for the teaching of sign language. In C. McBeath and R. Atkinson (Eds), Proceedings of the Second International Interactive Multimedia Symposium, 526-535. Perth, Western Australia, 23-28 January. Promaco Conventions.
http://www.aset.org.au/confs/iims/1994/qz/stuart.html
|
[ IIMS 94 contents ]
[ IIMS Main ]
[ ASET home ]
This URL: http://www.aset.org.au/confs/iims/1994/qz/stuart.html
© 1994 Promaco Conventions. Reproduced by permission. Last revision: 15 Feb 2004. Editor: Roger Atkinson
Previous URL 5 Oct 2000 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/iims/94/qz/stuart.html
