Detailed guide: Design your evaluation: evaluating digital health products

Public Health England

December 2
13:47 2020

This page is part of a guide to evaluating digital health products.

Decide what data to collect

You will need to work out how to collect the data you need for your evaluation. This may mean balancing the effectiveness of different data collection methods against the time and resources needed to carry them out.

Whether you want to run evaluation yourself or commissioning an external team, you should design your product to collect the data you need. Considering these practicalities will make your evaluation simpler to run:

Develop your product with evaluation in mind

This is best practice. Find out what you need to evaluate and make sure you can collect, record and store it in a consistent way over the period of the evaluation.

Consider evaluation when getting ethics approval

If you want to follow up with users, especially those who disengaged from your product, you need to have permission to email them. To associate data with the individual, you also need informed consent in advance.

Ensure the design of the product supports randomisation

The product should support randomisation between users and between events for a single user.

Types of data collection

Data collection through the digital product

Collecting data through the digital product is straightforward but means you cannot collect data on people who have stopped using the product. Some data collection from a digital product, like usage data, can be automatic. This is useful, but only some data can be collected in this way. Other data collection relies on users describing their own behaviour (self-report data), which can be inaccurate or biased.

Data collection must comply with the General Data Protection Regulation (GDPR) read guidance on GDPR. Its important to know that:

  • users may not always consent to data collection
  • GDPR requirements will introduce some bias in your data for example, the requirement for users of a website to accept cookies

App store data

App stores, or similar services, generally provide some statistics about an app, which you can access easily.

The number of people who have downloaded a product is a useful measure of impact. However, people often download an app but never use it, or only use it once. Downloads cannot tell you about longer-term engagement or whether a product is effective.

App stores give average user ratings (quantitative data) and access to individual reviews (qualitative data). These provide valuable feedback and will also influence whether someone will download an app. Reviews of your product and rival products can help to inform improvements.

However, popular apps are not always the most effective. People leaving reviews may also not be representative of most users.

Usage data

You can draw data from the usage of your digital product. The data you can collect will depend on how the product has been coded, so think about this when developing your product.

Commonly-available data includes:

  • when the user accesses the product
  • what parts of the product they viewed

Usage data is good for assessing engagement unlike downloads, it shows individuals are actually engaging with the product and how they are engaging with it.

Usage data cannot show whether an app is effective. However, usage data can be correlated with outcome data. This can show whether users who access the product more show a greater improvement in the outcome.

Data from smartphones

Many digital devices record data without the user having to do anything different (passive data collection). This reduces missing data and biases associated with who records data.

Smartphones typically have motion sensors and location data, which can be used, for example, to estimate physical activity. These measures are not perfect. For example:

  • users may not have the phone with them during exercise
  • users might disable data collection if it impacts on battery usage
  • users may not consent to data collection
  • phones cannot always differentiate between activities

Users may be more likely to consent to data collection if they see a benefit to it for example, if they are using an app for self-tracking purposes.

Data from wearable devices

Wearables offer more possibilities for passive data collection. Separate motion sensors may give more reliable data. Watch-like devices may directly measure things like heart rate and blood pressure (physiological indicators). Some devices target specific conditions, like continuous glucometers for diabetes.

Data collection can still be incomplete devices may require charging and upkeep and people may not always wear them. Data may also not be accurate for example, step count data is sometimes unreliable.

Self-tracking and self-report data

Many digital health products ask the user to input data (active data collection) as part of how they function. For example, if the app is used for self-tracking, users may track:

  • behaviours, such as physical activity or diet
  • how they feel, for example with a symptom tracker

Because it requires action from the individual, incomplete data collection is a common problem. You could still use this data for your evaluation and also ask users questions directly, separate to the products functions. For example, you can ask users:

Questions should be informed by your model of how your product works. Ask questions in a clear and neutral manner. It is worthwhile piloting your questions to check they work well.

This approach cannot tell you about the experience of non-users and usually cannot tell you about the lasting effects of your product. For example, does an app to promote physical activity still have an impact 5 years later?If someone stops using a product, this may not mean the product hasnt worked for them. For example, a Couch to 5K user may have stopped using the app because they became a confident and regular runner.

Linked data

Data about users of a digital health product or service may be collected for other reasons for example, when they attend NHS services in person. This data may be useful for your evaluation.

To link data from another source to your evaluation of a digital health product, you need to be able to identify the user of the product with the data collected elsewhere. This has legal and technical challenges, so discuss this with the relevant NHS body and consider what permissions you need.

Data collection outside the digital product

Sometimes the best way to collect data on an individual is to approach them directly, outside of and separate to the digital health product. For example, it may be possible to send them a questionnaire through contact details they provided or to meet them in person.

These approaches have problems with response rates but are better at reaching people who have disengaged from the digital product.

Sometimes you will want to collect more objective data from individuals. This might require specific testing, for example, measuring cotinine levels in saliva to objectively measure smoking behaviour.

Choose your design

Start by thinking about why you are doing an evaluation. Different types of evaluations answer different questions and may be required at different times. Different types of evaluation can complement each other. You rarely want to do just 1 evaluation over the life of a product.

Common reasons for doing evaluations include to:

  • check whether your product is effective
  • go through a regulatory process
  • inform the development of the product
  • understand users needs

There are some general points to consider when designing your study:

Choosing an approach

Choosing an approach often involves compromise. A more complicated study will require more time and resources but will give an answer in which you can have more confidence. A cheaper and quicker evaluation may be less rigorous. The Medi

Related Articles


  1. We don't have any comments for this article yet. Why not join in and start a discussion.

Write a Comment

Your name:
Your email:

Post my comment

Recent Comments

Follow Us on Twitter

Share This

Enjoyed this? Why not share it with others if you've found it useful by using one of the tools below: