Fact sheet 08-2021
How do we know that a diabetes app works?
The answer is that it is almost impossible to know because apps are evaluated in different ways. Some researchers in Norway have started to work on these criteria.
The number of mobile apps and online resources for people with diabetes are increasing steadily.
Studies have shown that those who use apps and other technology, often get health benefits in the short and long term. Patients gain more knowledge about their health, which allows them to better cope with the disease. Sharing one’s own health data with the doctor means that individuals could have treatments better tailored to them.
How to evaluate an app?
However, to find out if a mobile app has a positive effect on the patient, the app must be evaluated. The question is how.
Researchers do not have a standard way of doing this. They use different methods and criteria, which makes it difficult to compare the apps. Some researchers are concerned with what people think about using apps in general. Others look at the clinical impact, whether they can see changes in the overall health or in specific health outcomes.
Researchers at the Norwegian Centre for E-health Research (NSE) have conducted a systematic review of scientific publications from 2015 to 2018, which describes the evaluation of diabetes apps and online resources.
The overall aim of their project, funded by Helse Nord, is to create an overview of criteria for assessing the user-friendliness and effectiveness of apps and online resources for people with diabetes in Norway.
Patients are not always involved
The 31 international studies included in the review, had obtained responses from a total of 3689 people. In addition to people with diabetes, who accounted for only 12 % of those surveyed, the studies included feedback from health professionals (26 %), IT developers (13 %), researchers (10 %), parents and other family members (13 %) and others including experts and unspecified persons (26 %).
More than half of the studies dealt with diabetes apps. The rest evaluated websites or other diabetes-related content on the internet.
Mostly, the studies looked at the cognitive and clinical impact of using these technologies in people with diabetes. In addition, they gathered information about usability. But patients were not asked to evaluate security or privacy.
The researchers of the included studies rarely asked the users how they felt about using these technologies, and what they appreciated the most. Therefore, we can conclude that patients’ opinions must be made clearer in future research and development work. People with diabetes should not only be observed by researchers, but be actively involved in all aspects of the development of self-management apps and online resources.
Furthermore, researchers of the included studies were not particularly interested in topics such as sustainability and interoperability.
WHO checklist
In 2016, the WHO gathered several experts who agreed on some common criteria for evaluating technological tools. They then made a checklist for evaluating mobile apps.
The included criteria were:
- Sustainability and scalability
- Available to all users
- Ability to interact with other ICT solutions
- Possibility of customization
- Opportunity to recreate method and development
The criteria from the WHO are a good starting point. In addition, researchers must work to counteract socio-economic health inequalities. Standardization is necessary so that organizations and individuals who want to use the tools, can understand the technology.
The best way to evaluate the usefulness of apps and online resources is to involve patients all the way.
Reference:
Dillys Larbi et al.: Methods and Evaluation Criteria for Apps and Digital Interventions for Diabetes Self-Management: Systematic Review. J Med Internet Res. July 2020. DOI: 10.2196/18480
WHO checklist for evaluation of digital tools: Classification of digital health interventions: a shared language to describe the uses of digital technology for health.