GovWire

Detailed guide: Choose evaluation methods: evaluating digital health products

Public Health England

December 2
13:47 2020

This guidance is part of a guide to evaluating digital health products.

These resources look at 4 main types of study. If you are unfamiliar with them, read about each one before choosing your method. You can also combine methods.

1. Descriptive studies

Understand how things are going.

These types of studies describe what is going on. They can give you descriptive statistics (what proportion, how many, how much), or investigate correlations: relationships between variables.

Descriptive studies are weaker at providing evidence of cause and effect. They cant tell you what would have happened if you had done something else, for example, if a different product would have worked better.

Descriptive studies are often simple. For digital products, they can be quick and cheap if you can use data that is already being collected. Sometimes you will want to collect additional data specifically for the evaluation. This may cost more.

Most descriptive studies use quantitative methods, but some are qualitative or incorporate qualitative methods.

When to use them

Use them when:

  • your product is in use, to check it works and has no unintended consequences (ex-post evaluation)
  • your product is about to be in use and can be tested with a pilot group to see how it works in practice

Questions they answer

How many people are using your product?

How many people stay engaged with your product?

How often do they use it?

Do they like the product?

Descriptive study methods

Analysis of routinely collected data

Behaviour change techniques review

Clinical audit

User feedback study

2. Comparative studies

Check whether your product works.

To judge whether a digital health product or service is effective, you need to make a comparison to what the alternative would be. If this product didnt exist, what would have happened? The alternative might be nothing, an existing non-digital service, or a different digital product or service.

For example, a group of people who have some anxiety problems use your new app. Six months later, 30% report they no longer have anxiety problems. Does this prove that the app is effective? No. Some people recover from anxiety problems over time without outside assistance. You need to compare whether more people recovered using the app than would have without the app.

Comparative studies usually involve collecting quantitative data on individuals who are not given your product. This is more complex and expensive than a typical descriptive study.

The choice of comparison is important. Some comparisons are easier and cheaper to make, but may not be fair. You will have to find a balance between a very robust evaluation and what is practical and affordable. Some types of comparative study are very robust. They are recommended in the NICE Evidence Standards Framework for several types of digital product.

When to use them

Use them when:

  • you want to know how effective your product is, for example, in comparison to not using your product or to using another product
  • you are launching your product, or soon after (summative evaluation) and want to know how effective it is

Questions they answer

What is the difference in health outcomes between people using your product and people using a rival product?

What is the difference in health outcomes between people using 2 versions of your product?

What is the difference in health outcomes between people using your product now and how people did before your product was available?

Comparative study methods

Before-and-after study

N-of-1 study

Randomised controlled trial

A/B testing

3. Qualitative studies

Understand how people feel about your product.

Qualitative approaches can be good for understanding the thoughts and experiences of users of the digital health product, or of other stakeholders. Qualitative methods can give a richer explanation of what is happening. For example, a quantitative approach could describe patterns of disengagement with an app and what demographic factors predict disengagement, but a qualitative approach can describe why users stopped using the app. It could also focus on giving a voice to particular users who might otherwise be overlooked in evaluations. Qualitative studies may be more subjective than quantitative studies.

While qualitative studies can describe the range of views among participants, they are weak at saying how many participants hold different views. Results from qualitative studies are hard to generalise to other participants. The number of participants needed can be much lower than for other evaluation methods. However, the amount of information you get from each participant can be much higher, so usually more time is spent with each participant.

When to use them

Use them when:

  • you want a more in-depth understanding of users thoughts and experiences of your product
  • youre developing your product, to work out how to improve it (formative or iterative evaluation)
  • you have open-ended questions

Questions they answer

What was users overall experience of the product?

How did users feel during their journey through the product?

What do users want to see in a new version of the product?

Qualitative study methods

Ethnographic study

Focus group study

Interview study

Usability testing

4. Health economic studies

Understand what value for money your product provides and the cost implications of implementing it.

Health economic studies typically involve 2 broad types of economic analyses: economic evaluation and budget impact analysis.

Economic evaluation can help you assess the costs and effects of a digital health product and compare it with alternative options. It provides information about the relative effectiveness and cost-effectiveness of your product to help inform decisions about which digital health interventions to provide. Economic evaluation enables decision-makers to make efficient use of available resources for improving a populations health. Pursuing efficiency involves maximising the benefits of the available resources, not just reducing costs.

Budget impact analysis evaluates the affordability of a digital health product given the available budget constraints. This involves a comprehensive assessment of the costs and savings of implementing a new digital health product. Budget impact analysis usually complements value for money assessments and is often conducted after an economic evaluation.

Health economic studies can tell you whether the benefits of investing in your product exceed the benefits you would have got from investing the same amount in an alternative option. For example, if you are launching a new mobile app to facilitate remote GP consultations, economic evaluation will tell you whether your product provides most benefits for each additional pound invested in it compared to competing options. The alternative options could be other mobile apps or face-to-face appointments. It will also tell you about the likely cost implications of rolling out your product within the decision-makers available budget.

When to use them

Use them when:

  • your product is being launched or tested in a pilot study to help guide decisions about whether it should be adopted
  • you want to understand how your product compares with competing alternatives in terms of both benefits and costs
  • you need to know whether your product offers the most benefits given the available budget
  • you need a more comprehensive assessment of

Related Articles

Comments

  1. We don't have any comments for this article yet. Why not join in and start a discussion.

Write a Comment

Your name:
Your email:
Comments:

Post my comment

Recent Comments

Follow Us on Twitter

Share This


Enjoyed this? Why not share it with others if you've found it useful by using one of the tools below: