A Pervasive Multimodal Tele-Home Healthcare System

Abstract: This paper proposes a Human-centered Pervasive Computing System Model (HPC), a Layered Architectural Analysis and Design Method (LAAD) and a Waterfall Prototyping Process Model (WPP). Based on the HPC model and the LAAD method, a pervasive computing based multimodal tele-home healthcare system is designed and partly implemented using the Waterfall Prototyping Process. The design and implementation issues are discussed in more detail. Some testing results are presented.

Keywords: Pervasive Computing, Home Healthcare, Agent, Jini

Category: H.5.1, H.4.3, J.3, J.7

1 Introduction

With the increasingly aging population in the world, healthcare seriously challenges our society due to limited hospitals and other healthcare resources [Mynatt et al. 2000]. It is well-known that the healthcare cost crisis has troubled many developed countries, such as the US and Canada. Out-hospital healthcare such as tele-home healthcare may possibly solve the problem in the future.

Pervasive computing is considered as the next generation of IT technology which has great potential for many applications such as pervasive healthcare, smart automotive, smart home, intelligent education, intelligent work environment, seamless traveler service, pervasive mobile commerce, etc. Its "anytime, anywhere and human-centered" feature makes it a perfect technology for healthcare applications.

There is a great number of publications presenting work and research on pervasive computing applications in healthcare [Perry et al. 2004, Stanford 2002, Mihailidis et al. 2004]. With today's fast development of network and computer technology as well as the recent work on pervasive computing, Mark Weiser's vision of pervasive/ubiquitous computing [Weiser 1991] is coming true little by little. Since the end of the 20th century, this research has widely attracted researchers from institutions such as MIT [MIT Oxygen] and companies like IBM [IBM Project]. Recently, it has developed very quickly with the next generation information technology [Amor 2001, Satyanarayanan 2002, Saha et al. 2003, Abowd et al. 2000, Hansmann et al. 2003, Riva et al. 2005].

Lots of prototype systems are developed such as Seamless Messaging System by the National Research Council of Canada [Miao et al. 2002, Liscano et al. 1997] and Easy living System by Microsoft [Microsoft EasyLiving]. Soon these will come to the commercial market and some have already served us, although these kinds of systems are normally very complex.

This paper describes the design and implementation of a pervasive computing based multimodal tele-home healthcare system. It is based on previous work on two application oriented pervasive computing systems: a Seamless Messaging System (an intelligent workspace system) [Miao et al. 2002, Liscano et al. 1997] and a Pervasive Computing SmartLab System [Miao et al. 2005b]. From these two systems, this paper proposes a Human-centered Pervasive Computing Model, a Waterfall Prototyping Process Model and a Layered Architectural Analysis and Design Method for pervasive computing system analysis, design and implementation. The models and method are applied to analyze, design and implement the pervasive computing based multimodal tele-home healthcare system.

The structure of this paper is as follows: After this introduction, the Human-centered Pervasive Computing System Model (HPC) is presented in the second section. The third section describes the Layered Architectural Analysis and Design method (LAAD) as well as the Waterfall Prototyping Process Model (WPP). By applying HPqC, the system requirements and design issues of the Pervasive Computing based Multimodal Tele-home Healthcare System are discussed in section four. In section five, system and software structure and some implementation issues are described based on LAAD. Section six provides some testing results. Finally it comes to the conclusions.

2 Human-centered Pervasive Computing System Model (HPC) for Pervasive Computing System Analysis

To integrate all the pervasive computing system components as a whole so as to provide a high-level general view of pervasive computing systems, the Human-Centered Pervasive Computing System Model is proposed which is illustrated in Fig.1. This model is also introduced in [Miao et al. 2005a].

A pervasive system is basically considered as five component layers: Human Core Layer, Pervasive Human-Machine Interaction (HMI) layer, Pervasive Device layer, Pervasive Access layer and Pervasive Network layer. Pervasive Network layer as the system environment is the basis for the pervasive computing system. Pervasive Access layer is the bridge to connect the pervasive user device to the network environment. Certain different network access protocols such as WLAM and GSM need to be considered in this layer. Pervasive Device layer is for all devices that human core user directly or indirectly interacts no matter whether they are visible or invisible. Pervasive HMI layer considers the human and pervasive device natural interaction issues. As the model name indicates, the human core is the center of the model.

2.1 Pervasive Network Layer

This layer can be also called Universal Network/Unified Network/Pervasive Computing Environment. It includes all current or future networks and normally the network connects to the internet directly or indirectly. This layer considers software issues like Network Resource Management, Pervasive Middleware Platform, Network OS, etc. Server and gateway are important parts in this layer.

2.2 Pervasive Access Layer

This layer deals with pervasive network connection issues. Issues considered in this layer are Service Discovery and Management, Security and Privacy, Computing Paradigms such as Agent and Web Service, Integration of Physical and Information Space, Context Awareness for the network side, different network access protocols, etc. Server software structure and design is a key issue.

2.3 Pervasive Device Layer

This layer includes all user interactive devices no mater whether they are directly or indirectly interacted, visible or invisible, aware or unaware. Issues considered in this layer are Context Management, Sensors and Actuators, Smart devices, Device Software Structure and Design, Embedded System OS, etc.

2.4 Pervasive HMI (Human-Machine Interaction) Layer

This layer does not only mean Human-Computer Interaction, but also the human interactions with all network accessed devices such as with PDAs and various other information appliances. Issues considered in this layer are Context Awareness for the human side, Computing Paradigms for HMI, Integration of Human and Machine Space, Positioning and Tracking,Multi-modal interaction, User Interfaces (e.g. Situational / Tangible / Attentive), etc.

2.5 Human Core Layer

From the model's name, we can see that it clearly illustrates the human-centered concept for pervasive computing. The human core is surrounded by pervasive devices and pervasive networks. It is easy to understand the main ideas of pervasive computing are "anywhere, anytime and human-centered" concepts. Human/system requirements are major issues considered in this core layer based on various different applications. Applications considered for the Human Core Layer are such as health care, automotive, smart home, work environment, traveler service, wireless ticketing, mobile commerce, business application, etc.

3 Layered Architectural Analysis and Design (LAAD) method and Waterfall Prototyping Process Model

This section describes the LAAD method as well as Waterfall Prototyping Process Model [Miao et al. 2005b] based on Human-Centered Pervasive System Model.

For a real pervasive computing system development, the application requirements are first analyzed and then the system designed to meet the requirements. In mapping that to the HPC model, the requirement analysis is in the Human Core Layer and the design is in the four outer layers. How to design each layer and integrate them together are considered. As different issues are treated in each layer as different components which are shown in Fig.1, the component analysis and design are conducted. Then the component communication issues are considered.

Human Core Layer is for the human/system requirement analysis based on various applications such as healthcare, automotive, smart homes, work environment, traveler services, wireless ticketing, mobile commerce, business applications. For the four outer layer designs, each layer is considered as many components corresponding to different issues related to the layer. The four layer's components are presented as follows:

3.1 Pervasive Network Layer

This layer is mainly referred to the network hardware infrastructure and related software modules. The components in this layer are Gateways, Servers, Network Resource Management Module, Pervasive Middleware Platform, etc.

3.2 Pervasive Access Layer

The components in this layer are Service Discovery and Management, Security and Privacy, Computing Paradigms such as Agent and Web Service, Context Awareness for the network side, etc.

3.3 Pervasive Device Layer

This layer is mainly referred to the user interactive devices and their related software. The components in this layer are Sensors and Actuators, Smart devices, various user devices such as PDA, Context Management, Embedded System OS, etc.

3.4 Pervasive HMI (Human-Machine Interaction) Layer

The components in this layer are Context Awareness for the human side, Computing Paradigms for HMI, Positioning and Tracking, Multi-modal interaction, User Interfaces (e.g. Situational/Tangible/Attentive), etc.

For the whole system integration, the key issues are pervasive computing paradigm and the component communications. A key component in the whole system is the context aware computing component in the Pervasive Access Layer and Pervasive HMI Layer.

As for the engineering process the software engineering issues [Roger 2001, Sommerville 2001], waterfall model and prototyping model are normally suitable to pervasive computing system development. Evolutionary process models like the spiral model are normally not good as the pervasive computing system is not easily updated due to its complexities. One way for the pervasive computing system development is to integrate the waterfall model and the prototyping model into an integrated one which can be called Waterfall Prototyping Process Model (Fig.2).

The waterfall model process is used for a prototype development (starting step of prototyping model process) and then the prototyping model process is applied for the real application, that is, we use prototyping model but treat the product from waterfall model process as a prototype. In Fig.2, the first four steps (Requirements Analysis, System Design, System Implementation and Testing & Evaluation) are from the Waterfall Model. The last two steps (Adjustment & Refining, Final Evaluation & Finish) together with the first four steps are considered as a one-iteration Prototype Model. The process experiences only one prototype phase and then comes to the final user's product stage.

4 System Design Based on HPC and LAAD

This section describes how to use the Layered Architectural Analysis and Design Method to design the Pervasive Computing Based Multimodal Tele-home Healthcare System. First the whole Waterfall Model as Waterfall Prototyping Process Model's first four steps in Fig.2 is roughly used to build the system prototype. The method is LAAD, used as follows:

(1) Human Core Layer requirement analysis

The Pervasive Computing Based Multimodal Tele-home Healthcare System is mainly for health state monitoring, reporting, supporting and emergency rescuing at home or other out-hospital places. It should monitor the health states of some special people such as pregnant woman, or progress of some illness such as chronic diseases, support independent living of the elderly or other people with special needs, and respond to requests from the people. The monitoring information should be reported to the service center for their health state analysis which can also be used for medical or health information collection. If there is an emergent incident happening anytime/anywhere at home, it should be detected and immediately reported to relatives, community emergency center, corresponding hospital as well as closest ambulance center if the service is available.

(2) Outer four layer analysis and design

In order to meet the system requirements requested from the core layer mentioned above, firstly the system design should consider the HMI patterns, i.e., how to detect all necessary information in the Pervasive HMI Layer for the requirements, and how to respond to the requests from the user. There are three types of HMI information used: physiological signal, visual information and auditory information. To respond to the requests, visual and auditory information and reporting any emergent event are chosen.

Based on the Pervasive HMI Layer results, which devices to be considered in the Pervasive Device Layer are decided. The most useful devices for the HMI layer are computers including laptop and desktop at home or any service center. The monitoring devices for the system are physiological sensors, web cameras and microphones. Two physiological sensor equipments are used, wearable multi-physiological signal detection vest used during moving (Fig.3), and multi-physiological signal detection bed used during lying (Fig.4). For responding and emergency rescuing, speaker, display screen, communication devices such as mobile phone and PDA are considered.

The Light-weight Wearable Multi-physiological Signal Detection Vest can detect several physiological signals such as breath rate and pattern, heart rate, etc. The signals can be transmitted wirelessly to the receiver at home. The user can wear it like a normal vest to move, work, etc. With the WLAN environment, the user can move anywhere at home even in a big yard.

When the user goes to bed to rest during daytime or sleep at night, they may take off the vest without any concerns as a Super-sensitive Multi-physiological Signal Detection Bed is prepared for service. The bed is like a normal bed but can detect several physiological signals such as breath rate and pattern, heart rate, etc. The signals are recorded or transmitted as required.

After the Pervasive Device Layer, the network access manner is considered for all the devices. The devices can be connected to any types of network wire-line or wirelessly except physiological sensors. They should be wearable devices and connect to the network wirelessly. So a wireless environment is necessary at home.

Based on the above analysis, the Pervasive Network Layer structure is designed as WLAN at home (intelligent space environment), Community LAN, the internet and Chinese mobile telecomm network (China Mobile GSM/GPRS).

In general, the hardware structure of the Pervasive Computing Based Tele-home Healthcare System can be illustrated as in Fig.5.

5 Software Structure Based on LAAD

For the software design, the major work is in the Pervasive HMI Layer and the Pervasive Access Layer. Many software components are designed and implemented for each layer and integrated together. Software agent technology is used to implement many software components in the system. The system software structure is shown in Fig.6.

For Pervasive HMI Layer, the purpose for this layer is to get and understand user's information and respond to any requests from the user. For the healthcare system, it should monitor the user's health state information by physiological sensors, the physical state and activity information by visual /auditory sensors. When the user requests or needs any help, the system should actively offer the user the necessary help information by such as speaker, display screen, etc.

The Physiological software agent is designed to collect this kind of information such as EEG, EMG, EOG, ECG, Respiratory Patters, GSR, Blood Pressure and SPO. In many cases the user may not always wear the physiological sensor due to reasons such as forgetting or unwilling to wear, worrying to break or be injured if falling onto it.

As pervasive computing is an "anytime and anywhere" service, the system should also know some basic situation of the user when physiological information is not available. Visual agent and auditory agent are necessarily designed for that. Visual agent should detect the user from the background, track them when moving, understand activities to see if there are any abnormal events such as falling over, abnormal movement like crawl or skew, shaking like tic, long time lying somewhere such as the ground, at the door or elsewhere, where they should not be, mouth foaming or blooding, etc. The Auditory agent should detect and understand abnormal noises like serious cough, asthma, moaning, etc. When the user requests or needs any help, assistant agent is used to actively offer the necessary help information auditively by speaker or visually by display screen.

Four types of agent for the Pervasive HMI Layer are available: Video Context Perception Agent, Audio Context Perception Agent, Physiology Context Perception Agent, and HMI & Service Management Agent.

Video agent is used for detecting the user from the background, tracking them when moving and understanding the activity. Video agent (Fig.7) is a major part of the Video Context Perception module.

KQML can be used for the Communication module. In Fig.7, the Goal is defined by its task beforehand. The Goal Oriented Fusion uses Agent Oriented Pattern Recognition Concept.

Similarly there are Audio Context Perception Agent and Physiology Context Perception Agent. Their basic structure is the same as Video Context Perception Agent except the perception and processing module.

HMI & Service Management Agent is the major part of HMI & Service Management module. It is used to actively provide services to the user based on their needs and to process user's requests. Correct computing paradigms to fulfill the system requirements are chosen. For health state information report, the client-server paradigm is fine if the home server can be used for the service center client to obtain the information.

Otherwise another paradigm, such as message passing to send the information to the service center is used. For other requirements such as emergency rescuing, both client-server and messaging passing paradigms do not fit well. In the emergency case, the system should harmonize four sides (community emergency center, hospital service center, ambulance station and the relatives) to join in the rescue. The four sides have to work cooperatively for high efficiency. For example, the hospital should know and prepare the rescue before the ambulance comes to the hospital with the patient. Peer-to-Peer paradigms for their collaborative work is used.

For the Pervasive Access Layer, the major software components are context aware computing, resource management and security management. Context aware computing is mainly related to three modules/agents: Context Modeling Agent, Network Context Perception Agent and User Context Input Agent.

In Context Modeling Agent, when multi types of information are available, multimodal information fusion can be conducted to comprehensively analyze the user's health state, their needs, etc. Context Modeling Agent should obtain information such as user's activity (normal; abnormal), location (indoor such as at entry, corridor, living room, bed room, washing room, and dining room; outdoor such as backyard, front gate, balcony, and deck), status (ID, health state), etc.

The User Context Input Agent handles user's input information like sex, age, weight, health status, food preference, relatives, friends, social status, hobby, sport, habit, etc.

In Network Context Perception module, Derived Context Agent, which is closely related to Jini Lookup Service, [Flenner 2001, Jini Community] is designed to percept and manage network resources such as various devices. The agent should represent devices together with their detailed information such as computer with location and status, TV with volume and location, DVD Player with volume, Fridge with food item, Cell Phone with mode and volume, etc. Information for fixed non-Jini devices in the environment should be provided manually.

Jini devices such as mobile phones and laptops should discover and join the Jini lookup service automatically with the necessary information such as their location, mobility. The Jini environment can be shown in Fig.8.

All agents running in the Jini environment should communicate and work cooperatively as a whole. In the system, most agents are both Jini Client Agent and Jini Service Agent. They can request services offered by other agents via Jini Lookup Service, and they also provide services themselves.

Based on the above analysis, the major system ontology is shown in Fig.9. From the ontology, Resource Management Agent can be implemented. Legend: Class owl: Property rdfs: subClassOf

Based on the system ontology and by using OWL (Ontology Web Language) [Tao Gu et al. 2005, Masuoka et al. 2003, Masuoka1 2003], the Resource Management Agent realizes the major knowledge modeling and reasoning function of the system.

After the above prototype system is implemented and before its real application, Prototyping Model's step of user test and listening to the user (Waterfall Prototyping Process Model's fourth step in Fig.2) is used to see if the system is what the waterfall model expects, i.e., the final user satisfied system. This is the goal and the system may not be the final version. If not, the work goes to the Waterfall Prototyping Process fifth step, Adjustment & Refining, and then finishes the development.

6 Some Results

This section gives some testing results of the system. Fig.10 shows a video detecting, tracking and activity understanding example from Video Context Perception Agent. The user is jogging in the backyard and this context information is obtained by the system. The user may wear the Light-weight Wearable Multi-physiological Signal Detection Vest. If there is anything abnormal such as too high heart rate or strange breathing pattern detected, the speaker in the backyard will remind them. If there is anything urgent such as falling down and unable to stand up or even heatstroke, the Service Management Agent of the system will inform relatives or hospitals using mobile SMS message.

Fig.11 shows the real-time remote video monitoring function offered by HMI & Service Management Agent. When something happens at home or the user just wants to have a look at his home when out of town, they can use this function especially when informed of some events by mobile phone SMS message.

Fig.12 shows mobile phone SMS message informing function offered by HMI & Service Management Agent. Based on the user's setting and need, this function is activated. For example, unsafe urgent event happens such as when an intruder is detected which would seriously scare, interrupt or even injure the home-cared family member, or other dangerous people enter the home, and so on. If the home-cared family member has any emergent health problems, the HMI & Service Management Agent will inform the hospital service center, community emergency center, and relatives by SMS at the same time.

7 Conclusions

This paper proposes a Human-Centered Pervasive Computing System Model (HPC), a Layered Architectural Analysis and Design Method (LAAD) and a Waterfall Prototyping Process Model (WPP). Based on the HPC model and LAAD method, a Pervasive Computing based Multimodal Tele-home Healthcare System is designed and partly implemented. The design and implementation issues are discussed in more details. Some testing results are given. As this system has not been developed into a real application product yet, the Adjustment & Refining and Final Evaluation stages have not been applied to the development process so as to complete the whole Waterfall Prototyping Process. It is current and future work.

Acknowledgement

This work is supported by University Fund 2004SZ002, National 973 Key Research Program 2004CB318110 and Chinese Natural Science Foundation Project 60441002. Thanks many post-graduate and under-graduate students in our institutes for their contributions to the research work.

References

[Abowd et al. 2000] Gregory D. Abowd, Elizabeth D. Mynatt, Charting Past, Present, and Future Research in Ubiquitous Computing, ACM Transaction on Computer-Human Interaction, Vol.7, No.1, 2000, pp.29-58

[Amor 2001] Daniel Amor, Pervasive Computing: The Next Chapter on the Internet, Oct. 2001 (http://www.informit.com/articles/article.asp?p=165227)

[Flenner 2001] Robert Flenner, Jini and JavaSpaces Application Development, Sams Publishing, 2002

[Hansmann et al. 2003] Uwe Hansmann, et al, Pervasive Computing (second edition), Springer-Verlag Berlin Heidelberg, 2003

[IBM Project] IBM Project, http://www.developer.ibm.com/spc/pvc/index.html

[Jini Community] Jini Community, http://www.jini.org/

[Liscano et al. 1997] Liscano R., Impey R., Yu Q. and Abu-Hakima S., "Integrating Multi-Modal Messages across Heterogeneous Networks", IEEE International Conference on Communications, Montreal, Canada, June, 1997, pp.45-53

[Masuoka et al. 2003] Ryusuke Masuoka, et al, Ontology-Enabled Pervasive Computing Applications, IEEE INTELLIGENT SYSTEMS, SEPTEMBER/OCTOBER 2003, pp.68-72

[Masuoka1 2003] Ryusuke Masuoka1, Task Computing - the Semantic Web meets Pervasive Computing, Proc. 2nd Int'l Semantic Web Conf. 2003 (ISWC 03), Springer-Verlag, 2003, pp.866-881

[Miao et al. 2002] Zhenjiang Miao, Roger Impey, Baozong Yuan, "Seamless Messaging in a Multimedia Network Environment", Proc. of 6^th International Conference on Signal Processing (ICSP'02), Beijing, China, Aug. 26-30, 2002, pp.1031-1034

[Miao et al. 2005a] Zhenjiang Miao, et al, "Pervasive Computing Based Multimodal Tele-Home Healthcare System Design,International Conference on Smart Home and Health Telematics, July, 2005, Canada

[Miao et al. 2005b] Zhenjiang Miao, et al, "Pervasive Computing Engineering Concept, Method and Application to SmartLab System ", 2005 International Conference on Pervasive Systems and Computing (PSC'05), June, 2005, Las Vegas, USA, pp.161-164

[Microsoft EasyLiving] Microsoft EasyLiving Project, http://www.research,microsoft.com/vision

[Mihailidis et al. 2004] Alex Mihailidis, et al, "The Use of Computer Vision in an Intelligent Environment to Support Aging-in-Place, Safty, and Independence in the Home", IEEE Trans. On Information Technology in Biomedicine, Vol.8, No.3, 2004, pp.238-247

[MIT Oxygen] MIT Oxygen Project, http://www.oxygen.les.mit.edu/

[Mynatt et al. 2000] E.D.Mynatt, et al, "Increasing the opportunities for aging in place", ACM Conf. Universal Usability (CUU), Arlington, VA, 2000, pp.65-71

[Perry et al. 2004] Mark Perry, et al, "Multimodal and Ubiquitous Computing Systems: Supporting Independent-Living Older Users", IEEE Trans. On Information Technology in Biomedicine, Vol.8, No.3, 2004, pp.258-270

[Riva et al. 2005] G. Riva, et al (Eds), Ambient Intelligence, ISO Press, 2005

[Roger 2001] Roger S. Pressman, Software Engineering (fifth edition), McGraw-Hill, 2001

[Saha et al. 2003] Debahis Saha, et al, Pervasive Computing: A Paradigm for the 21^st Century, IEEE Computer, March 2003, pp.25-31

[Satyanarayanan 2002] M. Satyanarayanan, A Catalyst for Mobile and Ubiquitous Computing, IEEE Pervasive Computing, JANUARY-MARCH 2002, pp.2-5

[Sommerville 2001] Ian Sommerville, Software Engineering (6^th edition), Addison-Wesley Publishers Ltd, 2001

[Stanford 2002] Vince Stanford, "Using Pervasive Computing to Deliver Elder Care", IEEE Pervasive Computing, January-March 2002, pp.10-13

[Tao Gu et al. 2005] Tao Gu, et al, A service-oriented middleware for building context-aware services, Journal of Network and Computer Applications, Vol. 28, 2005, pp.1-18

[Weiser 1991] M. Weiser, "The Computer for the 21st Century," Scientific American, Sept., 1991, pp. 94-104; reprinted in IEEE Pervasive Computing, Jan.-Mar. 2002, pp. 19-25.