IIMS 94 contents
[ IIMS 94 contents ]

Functionality and implementation of a computer based, distributed multimedia patient record

Ulrich Arnold and Gerhard Peter
Medizinische Informatik
Universitat Heidelberg, Fachhochschule Heilbronn
This paper presents a computer based, distributed multimedia patient record. This system manages the entire patient record. It integrates traditional data with multimedia components. It is used for the management of all patients' data. Furthermore it enables the utilisation of patients' data in new ways. Initially the motivation for and aim of the project is presented. The computer based, distributed multimedia patient record is defined. Functions and supported areas of the system are outlined. The architecture of the system is explained. Finally the current status, some prototypes and next steps of the project are shown.

Motivation and aim of the project

Much data has to be archived for each in-patient in a hospital, eg. there are text, laboratory reports and results of imaging diagnostics eg. X-ray, computed tomography (CT) or nuclear magnetic spin resonance tomography (MR). Increasingly video sequences are archived, eg. for the documentation of endoscopy. This explains why medical data are composed of multimedia elements. A medical document can be regarded as a multimedia document. These items of information are stored in different ways and different places, that are normally far apart. It is difficult for the physician to obtain and use all the available information about a patient. This is a large problem especially in the diagnosis and treatment of cancer, where a lot of data and information are available on each patient. Gathering of information, analysis and evaluation of all data is very difficult and time consuming. A survey of all information, which is necessary and important for the physician, is often impossible. Therefore it would be most desirable to use a computer based multimedia patient record system in order to have access to all available information.

We are now developing an computer based, distributed multimedia patient record used for in-patients. This will allow patient records to be more widely and significantly used compared to the older method of conventional, paper records. Furthermore access to patient data is possible without delay, any and anywhere and even simultaneously with other accesses.

Definitions and functional aspects

A definition of the term computer based, distributed multimedia patient record can be achieved by explaining the four elements: patient record, computer based patient record system, distributed and multimedia.

A patient record (Dick, 1991) is the repository of information about a single patient. This information is generated by health care professionals as a direct result of interaction with a patient or with individuals who have personal knowledge of the patient. Traditionally, patient records are on paper and are used to store patient care data.

A computer based patient record system is an electronic patient record that resides in a computer system. The definition is restricted to systems whose purpose to manage the entire patient record. Therefore mechanisms for centralised knowledge, control, and organisational integrity are necessary.

The patient record is distributed in different ways.

Multimedia is interpreted and used in many different ways. It is popular and used as a buzzword, that gives products a good flavour. We use the definition proposed by Steinmetz:

Figure 1

Figure 1: Types of media in a computer based patient record

A multimedia system is characterised by the computer controlled, integrated processing, storage, representation, communication, creation and manipulation of independent pieces of information of various time dependent and time independent media.

In a patient record these different data types are:

Functional aspects

The most important design goals, functional aspects and advantages achieved by the use of an electronic patient record can be summarised as follows.

Access to patients' data is possible without delay at any time and even simultaneously from various locations, eg. doctors' rooms, operating theatre. All information about a patient is available, including pictures from medical imaging systems, digitised video sequences and sound recordings (Arnold, 1993).

Order entry and filling in of forms is supported in various ways for physicians and nurses. A physician can select and request a test from the application on his workstation. As soon as he orders a test, a quality and cost information concerning this test appears on the screen. Prescribed medications are further checked for interactions.

Data are entered only once into the record as close as possible to their origin. They are error checked. For example, it can be checked that value fall within expected ranges, they may be checked for spurious values, or for an unusual time variation.

Display of the information and reports is very flexible. It can be changed according to individual needs. Laboratory results can be displayed as tables, charts showing the relative distribution or as time, oriented diagrams showing the distribution over time.

Reminders support physicians through warnings and alerts.

Traditionally, a patient record is organised according to the sources and chronology of data, giving no help to the process of clinical problem solving. To support a physician's thought process a patient record should follow a suitable methodology as proposed by Weed (1969). Therefore the concept of the problem oriented patient record is realised. A problem list at the beginning of the record details both current and previous problems. An initial plan is derived for each problem. This is a plan for collecting further information to gain a diagnosis, for therapy or for information required on the Patient. Progress notes are attached to a problem. They are build up of a four part internal structure including the patient's subjective comments, objective findings on examination or laboratory testing, the physician's assessment of the patient status, and of the management plan that follows from that assessment. The system provides a guidance but it offers alternates and is modifiable within broad limits by the individual user.

The documentation is enhanced with documentation in a 3D-model. Beside the traditionally recorded vital signs, various images, video sequences and sound recording the physician uses a tool for sketching the size and location of tumours in a 3D-reconstruction. The physician can change scaling, clipping factor, lighting, reference point, viewing angle and he can determine which organs to display or not.

Findings from examinations or laboratory results are displayed. The system is used to contact a consultant. Here two different procedures are employed. In cooperative work both physicians discussing a problem simultaneously can see the same information on their display screen. In coordinated work a physician submits an annotated record to a consultant.

Flexible queries provide the physician access to archived data on a patient. The patient records can therefore be used for decision support through access to similar cases.

Computer assisted surgery (CAS) supports, for example. the designing and carrying out of a transplant for the reconstruction following surgery for throat cancer.

Interfaces enable the aggregation of data from various clinics for research purposes, eg. in centres for cancer research. By this means, confidentiality of patients' data is assured.

Research is supported by access to all patients' data for retrospective studies. Additionally, for retrospective studies, the selected criteria can be evaluated.

Figure 2

Figure 2: Functions of the system.

The first middle circle shows the areas of support concerning data of one patient. The second circle shows the areas of support concerning data of many or all patients. The outer circle shows the applications of the system. In hypermedia tutorial systems medical knowledge and experience are presented. The system is therefore used in the education of students and in the further training of physicians. For example, newly developed techniques in the surgery of throat cancer are presented and compared to traditionally used techniques using 3D-animations, video sequences, sound recordings and available information from selected patients.

For quality assurance, cases, results and outcomes can be compared. Statistics for procedures, costs and performance can easily be constructed.

System architecture

The user interface is designed to be very flexible according to different user needs. The application will be adopted without modifications to different platforms. A user interface management system (UIMS) has been used for the implementation. It enables user interface design and implementation in an object oriented manner independent of the window system. This results further in a clear distinction between interface and application. The interface can be adopted and changed without any modification to the applications being required. The UIMS models different users and user groups achieving flexible interfaces. The dialogue control directs human-computer interaction due to the user, user group and control files generated by the UIMS, while the discourse manager controls all the interface peripherals and takes part in the user workstation conversation.

The function selector as proposed by Silva (1992) receives a proper internal command. It manages the access to specific applications. The function selector uses knowledge about the user profile. If it recognises that the requested action is only available in another system, it uses the navigation aid to accomplish this task.

The navigation aid is an interface to external systems. It uses knowledge about external system commands and menu structures. The user is no longer burdened with having to know all the shortcuts and details of the external system. When changes are made to an external system, the navigation aid will absorb the impact and the user can continue to use the external system without retraining. The logical interface formats commands and menu calls in the required form. However the navigation aid it is not an adaptive software and can be replaced or updated as required. The communication interface controls communication peripherals and passes the commands of the logical interface the installed communication networks.

The major challenge designwise, will be the task of integration, that is, of building CPR systems using existing modules, subsystems, and components. A centralised control item called master record is needed, providing a single, integrated source to data. Users and applications must operate under the illusion that there is only one database, although the patient record is distributed within the network. This master record accesses the information in various databases or servers, laboratory systems or the so called life time record. The master record can be regarded as an abstraction of the physical data storage. This layer is responsible for accessing the information. For implementation a data manipulation language has been developed, which is adopted to the application requirements. The language is influenced by the new proposals for SQL 3 (Pistor, 1993) as well as by the proposals of the Object Management Group (OMG) (Cattell, 1993) for object oriented databases. Every access to data from each application occurs through this master record. This allows modifications in data storage and subsystems without any effects on the application.

The lifelong record contains all information of long term significance for a single patient. A removable, exchangeable disc is used as the storage medium. This lifetime record will be kept by the patient. Data can be exchanged between different hospitals and general practitioners. All kind of data will be summarised ranging from texts and pictures (CT, MR, etc.) to digitised video and sound sequences.

Various mechanisms are employed to assure data integrity and confidentiality. Audit trails occur at system, record and data field level. For user authentication new technologies will be used, eg. a "smart" card system. Data are classified in a three zone confidentiality model, where the innermost zone deals with extremely sensitive information. Physician and patient both have to agree to access, like the principle of using two keys, necessary to open a safety deposit box in a bank. The middle zone is the area where traditional confidentiality is necessary. Here a matrix mapping users and user groups to records and data fields is used. Finally, the outer zone, with less sensitive data, is accessible by all potential users.

All multimedia data are managed by a multimedia management system. This system is clearly designed so that it is not limited in further extensions or modifications. Moreover it should offer generality for media handling. It is cleanly modularised. This subsystem supports, as generally as possible, the display and input handling of multiple media.

Figure 3

Figure 3: Architecture of the distributed multimedia patient record

This system uses different tools which are similar for all media, eg. although editors are specific to a medium, they have all the same look and feel, eg. all window based, with a similar menu hierarchy and similar high level options.

Editors support the recording and manipulation of document content, structure or semantics. The storage of multimedia data covers the problem of saving a document on a longer term storage device. While it must be possible to store all the information on a document, how it is represented either internally or on a storage device may differ. The former is, in general, optimised for size (for example by using compression techniques), the latter is, in general optimised for speed, since it is used for processing purposes. Layout describes the process of presenting documents to be output from the processing environment to an environment accessible to human perception (eg. page layout). Whereas presentation describes the process of presenting documents for human perception, like displaying on screen or printing (which is a synonym for imaging as long as only visual media are concerned). As a visual retrieval tool, a browser allows as a quick scan through a set of items ( often the results of a former retrieval). Pictures may be displayed as icons, or as miniatures, the original item reduced in size or as descriptions, that are comments about the item's content. The browser is also used for following the hypermedia links.

Since the application is applied in a heterogeneous environment, ie. on various platforms ranging from PCs to high end workstations, under different operating systems and different window systems, data conversion is necessary. This applies from the conversion of text and graphics up to the problem of storing and displaying digitised video sequences.

Data storage of the multimedia data is based on various mechanisms. The multimedia database is based on an object oriented database system. This is owing due to the requirements for multimedia data, eg. special aggregation types and relationships are available. Three major groups of elements within the multimedia database can be defined:

  1. Proper multimedia objects such as images, sequences of sound or video. These objects may exist in various versions.
  2. Description data covers a wide spectrum of information, eg. registration data, description data or object structure description, and
  3. object related methods.
Other facilities are distributed, object oriented or relational databases as well as file structures accessed by NFS (Network File System, Sun) or AFS (Andrew File System), which is a part of the OSF Distributed Computing Environment. The lifetime record is used for archiving data with long term significance. An optical disk, which is kept by the patient and also used for data exchange with other institutions, is used for storing these data.


Up to now different parts of the systems and several prototypes have been realised and are presented in this chapter.

The use of computers opens up new ways in patient documentation. A system for the documentation of endoscopy results (Asprion, 1991), (Nohn, 1991) has been implemented in our institution which is based on the idea of drawing the size and location of the tumour in a three dimensional model of the head. The display of the model can be modified with the following operations:

A prototype system has been used for the evaluation of the decisions concerning user interface design for the patient record. The user interface has been implemented on a NeXT workstation with NextStep, gaining high productivity and rapid prototyping. The system is used to evaluate the requirements for the user interface, which were defined for the application of multimedia systems in medicine. The user sees and is able to use all functions of the system. The system is further used to study the feasibility integrating all media and data sources.

Another version of the user interface has been developed in a version which can be easily implemented on all available and installed operating and window systems, like Motif, Presentation Manager or MS Windows. Therefore, ISA Dialog Manager has been employed as an user interface management systems (UIMS), ensuring independence of the underlying window system. Further tools employed for the realisation are the graphical standard Phigs+, NFS, OSF DCE and NCS. Many of tools and peripheral devices have been tested for the integration of all media such as video and sound.

A central component is the so called master record used to achieve integration of all data from various sources, to assure data integrity, to manage redundancy, and to access data in the manifold sources like relational database systems, object oriented database systems or subsystems. It can be regarded as a virtual patient record with a central knowledge where the data reside and how to access it. For implementation purposes a data manipulation language has been developed, which is adopted to the need of the application. The language is influenced by the new proposals for SQL-3 as well as by the proposals of the Object Management Group (OMG) for object oriented databases. Every access to data from each application takes place through this master record. This allows modifications in data storage and subsystems without any effects on the application. For realising the master record lex and yacc has been used besides C++. The multimedia components of the patient records are all stored in the most suitable way and accessed through the master record.

Another tool to be used by the different applications is a multimedia toolset. The result is seamless integration of all different components in all applications. This toolset is responsible for tasks like digitising videos on machines with video digitisers, compressing them according to JPEG and MPEG, displaying MPEG videos on machines without a video board, mailing multimedia documents, browsing through different videos, pictures, supporting group work with a shared white board, etc. It has been built to have a uniform handling of multimedia data for all applications on all computers.

Different applications of the system are currently under development like the education component for example. It is used for the for transferring medical knowledge to students and physicians such as newly developed techniques in the surgery of throat cancer. It is a hypermedia authoring, presentation and learning environment, which combines a restricted access to selected patient records like video sequences of endoscopy, CTs etc. with the presentation of knowledge the in form of animations, texts and pictures.

Future goals

The work is carried on in various directions. Different components of system are completed and the prototypes will be evaluated in different hospital environments (clinic for ear, nose and throat, clinic for orthopaedics, clinics for teeth, mouth and jaw).

A new direction for the project is the development of a so called lifelong record. For each patient information of long term relevance is collected on an exchangeable disk. It is a summary of the lifetime of a patient. Data can be exchanged between different hospitals and general practitioners. All kind of data will he summarised, ranging from texts and pictures (CT, MR, etc.) to digitised video sequences and digitised sound sequences. Various tasks have to be solved in its development: specification of the content of the master record, data confidentiality, strategy for access, integration in existing information system and so on.


Arnold, U., Meyer, J. & Peter, G. (1991). Multimedia aspects in a documentation system for endoscopy of throat cancer. In Proc. of the 4th International Conference on Human-Computer interaction (HCI '91), ed. H.-J. Bullinger, pp. 384-388, Amsterdam: Elsevier.

Arnold, U., Meyer, J. & Peter. G. (1992). EDV-Unterstuetzung bei Diagnose, Therapie und Dokumentation von Tumorerkrankungen, Standoribestimmung. HNO-Kongress, 30.5 - 3.6.1992.

Amold, U. & Peter, G. (1993). A computer based, distributed multimedia patient record: Use of new technologies for computer based medical records. In Proceedings of the 1 lth international Congress of the European Federation for Medical Informatics, Jerusalem, April 18-22, 1993, A. Reichert, B. A. Sadan, S. Bengtsson, J. Bryant, U. Piccolo (Eds.), S. 585-590, London: Freund Publishing House.

Arnold, U. & Peter, G. (1993). A documentation system for endoscopy diagnosis of throat cancer and its integration into a computer-based, distributed multimedia patient record. In Proceedings 2nd European Conference on Engineering and Medicine, Stuttgart, April 25-29, 1993, Jan E. W. Beneken, Uwe W. Faust (Hrsg), S. 231- 232, Amsterdam: Elsevier.

Ball, M. J. & Collen, M. F. (1992). Aspects of the computer-based patient record. New York: Springer.

Bass, L. & Coutaz, J. (1992). Developing software for the user interface. Reading: Addison Wesley.

Blattner, M. M. & Roger, B. D. (Eds.) (1992). Multimedia interface design. Reading: Addison Wesley.

Booch, G. (1991). Object-oriented design with applications. Redwood City: Benjamin/Cummings.

Cattell, R. (Ed.) (1993). The object database standard ODMG-93. To appear in 1993.

Coad, P. & Yourdon, E. (1990). Object oriented analysis. Englewood Cliffs: Prentice Hall.

Coad, P. & Yourdon, E. (1991). Object oriented design. Englewood Cliffs: Prentice Hall.

Dick, R. S. & Steen, B. S. (Eds.) (1991). The computer-based patient record: An essential technology for health care. Washington DC: National Academy Press.

Duce, D. A., Gomes, M. R., Hopgood, F. R. A. & Lee, J. R. (Eds.) (1991). User interface management and design. Berlin: Springer.

Kejdahl, L. (Ed.) (1992). Multimedia: Systems, interaction and applications. Berlin: Springer.

Lun, K. C., Degoulet, P., Piemme, T. E. & Rienhoff, O. (Eds.) (1992). Proceedings of the 7th World Congress on Medical Informatics, Geneva, 6.-10.9.1992. Amsterdam: North Holland.

Meyer-Wegener, K. (1991). Multimedia Datenbanken. Stuttgart, B.G. Teubner.

Molich, R. & Nielsen, J. (1990). Improving a human computer dialogue. Communications of the ACM, 33, 3(March), 338-348.

Nielsen, J. (1990). Traditional dialogue design applied to modern user interfaces. Communications of the ACM, 33(10, October), 109-118.

Nohn, U. (1991). Konzeption und Teilrealisierung der Benutzerschnittstelle eines graphischen Dokumentationssystems fur Endoskopiebefunde. Diploma thesis Universitaet Heidelberg Fachhochschule Heilbronn, 1991.

Pistor, P. (1993). Objektorientierung in SQL3: Stand und Entwicklungstendenzen. Informatik Spektrum, 16(2), 89-94.

Rumbaugh, L, Blaha, M., Premerlani, W., Eddy, F. & Lorensen, W. Object-oriented modelling and design. Englewood Cliffs, NJ: Prentice Hall.

Shneiderman, B. (1992). esigning the user interface. Strategies for effective human computer interaction. 2nd Edition. Reading: Addison Wesley.

Silva, J. S. and Zawilski, A. J. (1992). The Health Care Professional's workstation: Its functional components and User Impact. In Marion J. Ball and Morris F. Collen (Eds.), Aspects of the computer based Patient Record. New York: Springer.

Steinmetz, R., Rueckert, J. and Racke, W. (1990). Multimedia-Systeme. In Informatik Spektrum, Band 13, Heft 5, Oktober 1990, Springer, Berlin, Heidelberg, New York. pp.280-282.

Steinmetz, R. and Herrtwich, R. C. (1991). Integrierte verteilte Multimedia-Systeme. Informatik Spektrum, 14(5), 249-260.

Weed, L. L. (1969). Medical record, medical education and patient care: The problem-oriented record as a basic tool. Chicago: Year Book Medical Publisher.

Authors: Ulrich Arnold
Dipl. Inf. med, assistant lecturer, Max-Planck-Str.
39 74081 Heilbronn, Germany
Tel: +49 7131 504 376 Fax: +49 71315 24 70
Email: arnold@sun1.rz.fh-heilbronn.de

Prof. Dr. Gerhard Peter
Max-Planck-Str. 39 74081 Heilbronn, Germany
Tel: +49 7131 504 330 Fax: +49 7131 5 24 70
Email: peter@fh-heilbronn.de

Please cite as: Arnold, U. and Peter, G. (1994). Functionality and implementation of a computer based, distributed multimedia patient record. In C. McBeath and R. Atkinson (Eds), Proceedings of the Second International Interactive Multimedia Symposium, 10-17. Perth, Western Australia, 23-28 January. Promaco Conventions. http://www.aset.org.au/confs/iims/1994/arnold.html

[ IIMS 94 contents ] [ IIMS Main ] [ ASET home ]
This URL: http://www.aset.org.au/confs/iims/1994/arnold.html
© 1994 Promaco Conventions. Reproduced by permission. Last revision: 19 Jan 2004. Editor: Roger Atkinson
Previous URL 25 Apr 2000 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/iims/94/arnold.html