Usability Evaluation of the Information System Used in Neuroscience Research Centres

Today, precise and reliable information is the bedrock for decision-making in healthcare systems [1]. Laws and regulations, research, human resources development, education, health service provision, and funding are essential in the development and enforcement of health related policies [2]. Information technology has revolutionised information processing precision and speed in many branches of human knowledge, including healthcare. All the aspects of work in an organisation need accessible technology for solving problems or performing tasks [3]. Information technology is a supporter of healthcare system processes in societies [1]. The healthcare sector, especially healthcare centres, health information systems for collecting, storing, processing, and exchanging a large volume of produced information [4] and regard it as a major contributor to the quality of healthcare. The health information system provides a basis for decision-making; collects, analyses, and stores data; ensures that the data are of high-quality, relevant, and up-to-date; and converts data into information for health related decision-making [5]. These are computerised systems designed for facilitating health information in healthcare centres to improve healthcare quality [6]. Health information systems are used as a tool to facilitate healthcare and administrative activities. In addition to reduce errors, increasing speed and precision and improving user satisfaction, these systems reduce healthcare costs through coordination and improvement of healthcare quality [7-9].

The diseases of the nervous system are the main cause of disability-adjusted life years due to mortality or disability [10]. Based on the World Health Organisation (WHO) report, millions of people worldwide are affected by the diseases of the nervous system. Annually, more than six million people die due to strokes, and more than 80% of these deaths occur in low and moderate-income countries. Furthermore, over 50 million people suffer from epilepsy worldwide. It has been estimated that, globally, 47.5 million people have dementia, with 7.7 million new cases per year. Alzheimer’s disease is the most common cause of dementia and may be its main cause in 60-70% of cases. Over 2.5 million of the world population suffer from Multiple Sclerosis (MS), the majority of whom are women. In addition, more than 10% of the world population suffers from migraines [11]. In Iran, due to the increasing trend of population ageing, Alzheimer’s disease is on the rise [12]. The prevalence of MS is also increasing in Iran, especially among the young and the women [13]. Due to the rising trend of the disease of the nervous system and their complications [12], there is a necessity for establishing research centres. To develop medical knowledge, these centres help improve the health of society through optimising research and updating clinical services. The use of information systems in these centres facilitates the achievement of this goal and marks the path towards modern medicine [14].

With regard to the positive effect of information systems on patient care and the continuous use of these systems by doctors and nurses, their evaluation is essential for ensuring the satisfaction of users’ and healthcare organisations’ information needs, conducting epidemiological research, managing health information, preventing rework, promoting the quality of care, reducing costs, determining user satisfaction level, system effectiveness and efficiency, identifying these systems’ weaknesses and strengths, and presenting solutions for system improvement [15,16].

If information systems do not satisfy the users’ needs, they will be abandoned despite their successful preliminary acceptance [17]. Researchers have concluded that there is a need to understand users’ intent to continue using the system [18]. There are several methods for evaluating different aspects of information systems. Attending all the aspects of health information systems and their all-inclusive promotion will greatly affect the services of healthcare organisations and, eventually, promote patients’ health [19]. Usability is a quality criterion for information systems, which is evaluated by assessing the User Interface (UI) [20]. Researchers have concluded that usability problems, which are associated with poor design, difficult learning, and complexity of use of information systems and causes problems for user interaction with the system, can negatively affect the users’ time management and efficiency, leading to confusion, and eventually, system failure. Moreover, not using/no intent for continuous use of information systems demonstrates the system’s poor usability, which, in turn, leads to undesirable system efficiency. Heavy costs are paid for system design, but the system will be abandoned if it has no users, and new information systems have to be purchased and installed by the Institution [21,22].

System usability is the degree/level of a system’s assistance to users in performing their tasks. This feature guarantees the simplicity, naturalness, adjustment, and support for users against a heavy workload in the system. With these features, systems help users to quickly and rapidly perform tasks with minimum mental effort, and thus meet their needs [23].

With respect to the importance of information systems in neuroscience research centres, and the necessity of their usability evaluation, the present study was aimed to evaluate the usability of the information system used in Neuroscience Research Centres of hospitals affiliated with SBMUS, Tehran, Iran.

Materials and Methods

This was a descriptive study conducted in July 2020 at SBUMS. The study population was the information system of Neuroscience Research Centres at hospitals affiliated to SBUMS. Before starting the study, ethical considerations such as obtaining informed consent, anonymity, confidentiality, and the participants’ freedom to withdraw from the study were taken into account. This study was discussed, reviewed, and approved by the SBMU Ethics Committee (ethics code: IR.SBMU.RETECH.REC.1399.635).

As this research aimed to investigate the usability of the information system at neuroscience research centres, among the centres affiliated to the SBUMS, only hospitals with neuroscience diseases departments were selected as the research sample (of which only two hospitals met this requirement which were Educational hospitals affiliated with SBMUS). To gather the samples, the census method was used as the population under study was limited. Twenty five people participated in this study (18 nurses, one physician, one accountant, one at humanities, one at information technology, one at management, one at medical engineering, one at software engineering level).

Inclusion criteria: Inclusion criteria in this study were Neurology staff (physicians, nurses and secretaries) who were users of information systems.

Exclusion criteria: Exclusion criteria in this study were staff of other wards of the hospital.

Study Procedure

First, the objectives were specified, and then questions dealing with these objectives were extracted based on the questionnaires presented in similar papers [24-26]. Information systems cannot be evaluated without analysing the users’ requirements of the system. Moreover, usability problems pertaining to poor design, difficult learning, and the complexity of using the information system complicate users’ interaction with the system. These problems can impact the staff’s decision-making, time management, efficiency, lead to fatigue and confusion, and eventually reduce information system efficiency [19,21,22]. Therefore, for information system evaluation, authors selected several criteria out of the main criteria of usability. Based on the ISO/DIS 9241-11 standard, usability refers to the “extent to which a product can be used by specific users to reach pre-determined goals in a specific setting with effectiveness, efficiency, and personal satisfaction of the users” [27], and is a more comprehensive concept than ease of use. Furthermore, according to Nielsen J, usability has several components, including memorability, learnability, efficiency, satisfaction, and errors [28]. Thus, these criteria were examined in the information systems.

The data collection instrument was a 32-item questionnaire based on the Usefulness, Satisfaction, and Ease of Use (USE) [29] and IsoMetrics questionnaires [30]. This combined questionnaire comprised of five domains of usefulness, ease of use, memorability, user satisfaction, and learnability, with items scored on a 7-point Likert scale. The other questionnaire was USE questionnaires which is a 30-item questionnaire that measures system usefulness, ease of use, learnability, and user satisfaction on a 7-point Likert scale [31]. The 75-item IsoMetrics questionnaire which is a valid and reliable tool for user-centered evaluation was also employed. IsoMetric questionnaire, seven principles of suitability for the task, self-descriptiveness, controllability, error tolerance, suitability for individualisations, suitability for learning, and conformity with user expectations were examined [15]. These questionnaires were combined based on the opinions of experts (faculty members with 10 years of work experience in evaluating information systems). The validity of the questionnaire was assessed through content validity that was confirmed based on the opinions of 10 faculty members (seven experts of information management, and three with a PhD in medical informatics), and reliability was assessed based on Cronbach’s alpha value (0.88).

This questionnaire had 32 items, including Usefulness (eight items), Ease of Use (11 items), Ease of Learning (three items), and User Satisfaction (seven items) from the USE questionnaire, while memorability had 3 items of which two are taken from the IsoMetrics Usability questionnaire and one from the USE questionnaire [Annexure-1]. These items ranged from strongly disagree (1) to strongly agree (7), with an option for no opinion. These were measured on a 7-point Likert scale and received scores from 1 to 3 (undesirable), 3.1 to 5 (relatively desirable), and 5.1 to 7 (relatively desirable). The participants did not choose no opinion for any question.

Statistical Analysis

The data collected via the questionnaires were inputted to Statistical Package for the Social Sciences (SPSS) version 26.0, analysed by descriptive statistics, and presented in tables and diagrams. To evaluate the usability of the information systems, the scores of each item were first calculated from the frequency of the answers to the items, the assumed numerical value of each item and the weight factor of one, and the final item score was reported in percentages as the average percentage of each score was also determined.

Results

[Table/Fig-1] shows the calculation of Cronbach’s alpha to determine the reliability of the questionnaire.

[Table/Fig-1]:

Calculating Cronbach’s alpha for the dimensions of the questionnaire.

Variables	Cronbach’s alpha	Number of items
Usefulness	0.84	8
Ease of use	0.66	11
Ease of learning	0.81	3
Satisfaction	0.85	7
Memorability	0.42	3
Total reliability	0.88	32

The majority of the participants were women (76%). The age group of 25-29 years had the highest frequency, while the age group of 20-24 years had the lowest frequency. Moreover, 72% of the participants had a BS degree, and 4% had a PhD. Most of the participants were nurses (72%) and 47.3% of participants had 1-10 years of work experience, the majority of whom had studied nursing (72%). Two hospitals which were Educational hospitals affiliated with SBMUS, here referred to as Hospital A and Hospital B, respectively.

Based on the findings, the mean total score of the information system was 6.14±1.6 in Hospital A, and 5.73±1.04 in Hospital B for the criterion of usefulness. The highest mean score for the two hospitals belonged to the question “It makes the things I want to accomplish easier to get done” (6.23±0.72). As for ease of use, the mean score of this criterion was 4.71±1.53 in Hospital A and 4.82±1.23 in Hospital B. The highest mean score for the two hospitals belonged to the question “It is easy to use” (5.41±1.99).

Moreover, the mean score of learnability was 5.69±1.47 in Hospital A and 5.89±0.85 in Hospital B. For the two hospitals, the highest mean score belonged to the question “learned to use the system quickly” (5.92±1.11). For the user satisfaction criterion, the mean score was 4.95±1.43 for Hospital A and 4.84±1.42 for Hospital B. For the two hospitals, the maximum mean score belonged to the question “would recommend the use of the system to a friend” (5.75±1.05), followed by the memorability criterion with the mean of 4.66±1.61 and 5.79±0.93 for Hospitals A and B, respectively. For the two hospitals, the highest mean score was 5.84±0.98 for the question “It is easy for me to relearn how to use the software after a lengthy interruption”.

The scores of the criteria in [Table/Fig-2] were also examined, and the following scores were obtained: usefulness (5.93), user satisfaction (4.89), ease of use (4.76), learnability (5.79), and memorability (5.22).

[Table/Fig-2]:

The mean scores of the criteria based on the information system users’ views at the hospitals affiliated with SBMUS.

Discussion

The present study evaluated the usability of the information system used in neuroscience research centres of hospitals affiliated with SBMUS. The evaluation was performed by using USE and IsoMetrics questionnaires. The USE questionnaire assesses four parameters of usefulness, ease of use, learnability, and user satisfaction. Based on the findings, 31.96% of the users at A and B Hospitals attested to the information system’s usefulness; more than 82% of the users selected somewhat agree to totally agree, and the mean of both hospitals for this criterion was 5.9. A study by Pangestu H and Karsen M on a Learning Management System (LMS) showed that the system’s usefulness is 5.13, which was excellent [32]. Paulino T et al., also evaluated the usefulness of an integrated game-based system for the elderly, and the results demonstrated a score of 5.3 for this criterion [33]. The findings of Prastyo D and Bakhtiar MY showed the relatively desirable usability of the Human Resources Information System (HRIS) with a mean of 3.67, which reached 4.04 after development and improvement. Moreover, the usefulness of the system was 3.99, which reached 4.21 upon development and improvement [34]. In a study by Faria TVM et al., to examined the usability of a web-based system by using the USE questionnaire, the usability of the system was 5.6 and at an optimal level. Among the questions, “Does the system meet the users’ expectation?” had the lowest score (4.8%) [26].

From the questions dealing with usefulness in this questionnaire, the question “Does the system save time?” had the highest score (100% agreement) in Hospital A. Based on the findings, it seems that the designers of information systems used in the research centres of hospitals affiliated with SBUMS have optimally attended to the time-saving nature and ease of use of the system, and their designed systems have reasonably ensured these features.

As for the ease of use of the information system, 43.65% of the users at Hospitals A and B somewhat agreed to this criterion; ease of use had the mean of 4.76 in the two hospitals, which indicates a good level. In the study by Prastyo D and Bakhtiar MY, the mean score of ease of use of the HRIS was 3.41, which reached 3.61 after improvement [34]. Moreover, Jarvis JM et al., reported that the ease of use of an electronic health application to be 4.4 [35]. In the study by Faria TVM et al., a mean of 5.76 was obtained for this criterion, showing an excellent level [26]. Based on present study findings, it seems that the designers of the information systems of the research centres of SBUMS have paid sufficient attention to the ease of use of the system.

An evaluation of the ease of learning of the information system revealed a mean score of 5.79 in the two hospitals. Pongthananikorn S evaluated the usability of a website based on the electronic Personal Health Records (ePHRs) of patients with renal diseases and concluded that the mean ease of learning of this system was 5.63 [35].

Furthermore, Martono KT et al., evaluated the usability of a patient health monitoring system and reported an ease of learning score of 3.92 [1]. Based on the results, it seems that the designers of the information systems used at the research centres of SBUMS have paid particular attention to the ease of learning, especially the speed of learning, and their designed system can ensure this criterion. Also, attention to this criterion has enhanced the level of usability of the system.

An assessment of the user satisfaction with the information system revealed a mean of 4.89 in the two hospitals. According to Jarvis JM et al., user satisfaction with an electronic health application was 4.2 [35]. From the questions dealing with this criterion, the highest level of agreement belonged to “Do you recommend the use of the system to your friends?”. Thus, it seems that the designers of the information systems of the research centres of SBMUS have paid little attention to user satisfaction.

IsoMetrics was employed to evaluate the memorability of the information system. The results indicated that this criterion had a mean of 5.22 in the two hospitals, which shows a good level. Munaiseche C and Liando O assessed the usability of an expert system and reported that the memorability of the system is 4.14 [37]. Accordingly, it seems that the designers of the information systems used in the research centers of SBMUS have attended well to the memorability of the system.

Limitation(s)

The number of the subjects who filled the questionnaire was relatively less. Further studies with a larger sample size should be conducted in future.

Conclusion(s)

Based on the findings of the present study, it can be concluded that the health information system has a good and acceptable to excellent level in terms of usability criteria, based on the dimensions of USE and IsoMetrics questionnaires. Of the examined criteria, usefulness and learnability achieved the highest score, indicating the suitable design of the system in terms of these dimensions. On the other hand, ease of use had the lowest score, showing the poor user design of this information system. Thus, to achieve an excellent level of information system usability in Neuroscience Research Centres, attention should be paid to all the dimensions of information system usability.

[1]. Martono KT, Nurhayati OD, Widodo AP, The Evaluation of Child’s Health Monitoring System Using the Usability Testing Approach 2018 5^th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE) 2018 IEEE10.1109/ICITACEE.2018.8576933  [Google Scholar]  [CrossRef]

[2]. Lin C, Lin IC, Roan J, Barriers to physicians’ adoption of healthcare information technology: An empirical study on multiple hospitals J Med Syst 2012 36(3):1965-77.10.1007/s10916-011-9656-721336605  [Google Scholar]  [CrossRef]  [PubMed]

[3]. Haried P, Claybaugh C, Dai H, Evaluation of health information systems research in information systems research: A meta-analysis Health Informatics J 2019 25(1):186-202.10.1177/146045821770425928441893  [Google Scholar]  [CrossRef]  [PubMed]

[4]. Haux R, Health information systems- from present to future? Methods Inf Med 2018 57(S 01):e43-45.10.3414/ME18-03-0004  [Google Scholar]  [CrossRef]

[5]. Sirintrapun SJ, Artz DR, Health information systems Clin Lab Med 2016 36(1):133-52.10.1016/j.cll.2015.09.01226851670  [Google Scholar]  [CrossRef]  [PubMed]

[6]. Berg M, Implementing information systems in health care organisations: Myths and challenges Int J Med Inform 2001 64(2-3):143-56.10.1016/S1386-5056(01)00200-3  [Google Scholar]  [CrossRef]

[7]. Su KW, Liu CL, A mobile nursing information system based on human-computer interaction design for improving quality of nursing J Med Syst 2012 36(3):1139-53.10.1007/s10916-010-9576-y20827569  [Google Scholar]  [CrossRef]  [PubMed]

[8]. Malliarou M, Advantages of information systems in health services Choregia 2009 :5  [Google Scholar]

[9]. Devine EB, Hansen RN, Wilson-Norton JL, Lawless NM, Fisk AW, Blough DK, The impact of computerised provider order entry on medication errors in a multispecialty group practice J Am Med Inform Assoc 2010 17(1):78-84.10.1197/jamia.M328520064806  [Google Scholar]  [CrossRef]  [PubMed]

[10]. Collins TR, Neurologic diseases found to be the largest cause of disability worldwide Neurology Today 2017 17(22):01-32.10.1097/01.NT.0000527316.80068.88  [Google Scholar]  [CrossRef]

[11]. Prince M, Bryce R, Ferri C, World Alzheimer Report 2011: The benefits of early diagnosis and intervention 2018   [Google Scholar]

[12]. Aajami Z, Kebriaeezadeh A, Nikfar S, Direct and indirect cost of managing Alzheimer’s disease in the Islamic Republic of Iran Iran J Neurol 2019 18(1):07-12.10.18502/ijnl.v18i1.940  [Google Scholar]  [CrossRef]

[13]. Etemadifar M, Sajjadi S, Nasr Z, Firoozeei TS, Abtahi SH, Akbari M, Epidemiology of multiple sclerosis in Iran: A systematic review Eur Neurol 2013 70(5-6):356-63.10.1159/00035514024192707  [Google Scholar]  [CrossRef]  [PubMed]

[14]. Ghorbani NR, Asadi F, Rangraz MH, Investigation of clinical information system in gastrointestinal and liver diseases research centers: Challenges and solutions Crescent J Med Biol Sci 2018 5(4):279-84.  [Google Scholar]

[15]. Hamborg KC, Vehse B, Bludau HB, Questionnaire based usability evaluation of hospital information systems Electron J Inf Syst Eval 2004 7(1):21-30.  [Google Scholar]

[16]. Bürkle T, Ammenwerth E, Prokosch HU, Dudeck J, Evaluation of clinical information systems. What can be evaluated and what cannot? J Eval Clin Pract 2001 7(4):373-85.10.1046/j.1365-2753.2001.00291.x11737529  [Google Scholar]  [CrossRef]  [PubMed]

[17]. Khubone T, Tlou B, Mashamba-Thompson TP, Electronic health information systems to improve disease diagnosis and management at point-of-care in low and middle income countries: A narrative review Diagnostics 2020 10(5):32710.3390/diagnostics1005032732443856  [Google Scholar]  [CrossRef]  [PubMed]

[18]. Ketikidis P, Dimitrovski T, Lazuras L, Bath PA, Acceptance of health information technology in health professionals: An application of the revised technology acceptance model Health Informatics J 2012 18(2):124-34.10.1177/146045821143542522733680  [Google Scholar]  [CrossRef]  [PubMed]

[19]. Khajouei R, Zahiri Esfahani M, Jahani Y, Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems J Am Med Inform Assoc 2017 24(e1):e55-60.10.1093/jamia/ocw10027497799  [Google Scholar]  [CrossRef]  [PubMed]

[20]. Farzandipour M, Nabovati E, Zaeimi GH, Khajouei R, Usability evaluation of three admission and medical records subsystems integrated into nationwide hospital information systems: Heuristic evaluation Acta Inform Med 2018 26(2):133-38.10.5455/aim.2018.26.133-13830061787  [Google Scholar]  [CrossRef]  [PubMed]

[21]. Chin W, Lee M, A proposed model and measurement instrument for the formation of IS satisfaction: The case of end-user computing satisfaction CIS Proceedings 2000 :57  [Google Scholar]

[22]. Limayem M, Hirt SG, Cheung CM, How habit limits the predictive power of intention: The case of information systems continuance MIS quarterly 2007 :705-37.10.2307/25148817  [Google Scholar]  [CrossRef]

[23]. Marcus A, Return on investment for usable user-interface design: Examples and statistics User Experience Magazine 2002 1(3):25-31.  [Google Scholar]

[24]. Ahmadi M, Shahmoradi L, Barabadi M, Hoseini F, Usability evaluation of hospital information systems based on isometric 9241 Hakim Research Journal 2011 [cited 2021September09] 13(4):226-33.Available from: https://www.sid.ir/en/journal/ViewPaper.aspx?id=244372  [Google Scholar]

[25]. Mousavi A, Hemati M, Kayyal S, Shamshiri A, Usability evaluation of hospital information systems based on isometric standard from the viewpoint of usersin education and treatment centers affiliated to Qom University of Medical Sciences, Iran Muq-Journal 2014 8(4):58-63.  [Google Scholar]

[26]. Faria TVM, Pavanelli M, Bernardes JL, Evaluating the usability using USE questionnaire: Mindboard system use case International Conference on Learning and Collaboration Technologies 2016 Springer  [Google Scholar]

[27]. Bevan N, Carter J, Earthy J, Geis T, Harker S, New ISO standards for usability, usability reports and usability measures International Conference on Human-Computer Interaction 2016 Springer10.1007/978-3-319-39510-4_25  [Google Scholar]  [CrossRef]

[28]. Nielsen J, Usability engineering 1993 Academic Press:6810.1016/B978-0-08-052029-2.50009-7  [Google Scholar]  [CrossRef]

[29]. Lund AM, Measuring Usability with the USE Questionnaire STC Usability SIG Newsletter 2001 8:2  [Google Scholar]

[30]. Gediga G, Hamborg KC, Düntsch I, The IsoMetrics usability inventory: An operationalization of ISO 9241-10 supporting summative and formative evaluation of software systems Behav Inf Technol 1999 18:151-64.Available from: https://www.isometrics.uni-osnabrueck.de/index.htm10.1080/014492999119057  [Google Scholar]  [CrossRef]

[31]. Johnson C, Johnston D, Crowle P, EHR usability toolkit: A background report on usability and electronic health records 2011 Rockville, MDAgency for Healthcare Research and Quality  [Google Scholar]

[32]. Pangestu H, Karsen M, Evaluation of usability in online learning 2016 International Conference on Information Management and Technology (ICIMTech) 2016 IEEE10.1109/ICIMTech.2016.7930342  [Google Scholar]  [CrossRef]

[33]. Paulino T, i Badia SB, Cameirão M, Usability evaluation of an integrative exergaming system for the senior population 2019 5^th Experiment International Conference (exp at’19) 2019 IEEE10.1109/EXPAT.2019.8876582  [Google Scholar]  [CrossRef]

[34]. Prastyo D, Bakhtiar MY, Development of the Human Reis Information System (HRIS) Based on Usability Analysis with Usefulness, Satisfaction, Ease to Use (USE) Questionaire and Cognitive Walkthrough Methods Bit-Tech 2019 1(3):150-62.10.32877/bt.v1i3.77  [Google Scholar]  [CrossRef]

[35]. Jarvis JM, Gurga A, Greif A, Lim H, Anaby D, Teplicky R, Usability of the Participation and Environment Measure Plus (PEM+) for client-centered and participation-focused care planning Am J Occup Ther 2019 73(4):7304205130p1-7304205130p810.5014/ajot.2019.03223531318677  [Google Scholar]  [CrossRef]  [PubMed]

[36]. Pongthananikorn S, Development of a chronic kidney disease knowledge website with electronic personal health records for patients Thai Journal of Pharmaceutical Sciences 2016 :40  [Google Scholar]

[37]. Munaiseche C, Liando O, Evaluation of expert system application based on usability aspects IOP Conf Ser Mater Sci Eng 2016 :12801200110.1088/1757-899X/128/1/012001  [Google Scholar]  [CrossRef]