The Business Case For Rigorous Evaluation Of Mobile Health Apps

The mobile health app industry (also known as mHealth) is booming. App developers are hoping to engage users in health behavior change, including weight loss and smoking cessation, and are also targeting mental health concerns such as anxiety. Although many of these apps claim to be science-based, rigorous data on their efficacy or outcomes are scant. Researchers have developed numerous frameworks, tools, and criteria for evaluating them, and consumer-based evaluations such as app store star ratings are misleading at best. Without rigorous evaluation, it will be impossible to distinguish apps with the potential to transform human health from those that more closely resemble 21st-century snake oil.

The low rate of rigorous evaluation of mobile health apps is at least partially due to their regulatory status. Unlike other interventions that pose higher risks for adverse effects, such as pharmaceutical drugs or medical devices, many mobile health apps fall under a regulatory gray area. The Food and Drug Administration (FDA) has stated that it “intends to exercise enforcement discretion” for many mobile health apps focused on behavior change. However, as long as no specific claims are made about the tools’ ability to diagnose or treat disease, mobile health apps are generally not considered medical devices and do not need to demonstrate safety or effectiveness in clinical trials. In practice, many apps claim to have been evaluated and use scientific language to describe their efficacy. Yet, they almost never have published research to back them up.

The rarity of rigorous evaluation of mobile health apps is also due to the fact that it is typically app developers—not health systems, scientists, or clinicians—who are leading these efforts to digitize health behavior change. In cases where health systems and clinician researchers are leading the effort, incentives seem more likely to promote rigorous evaluation, even though efficacy data are limited in these environments. For example, numerous freely available apps have been developed by Veterans Affairs health researchers, but efficacy and effectiveness data are still scarce.

If an app can potentially survive or thrive without undergoing rigorous evaluation of outcomes, developers may not be willing to invest the time and resources, or take the risk of disappointing results—especially if they are more focused on hitting key performance indicators such as numbers of users and time spent on the app. Nonetheless, the business case for rigorous evaluation of mobile apps is becoming more relevant thanks to recent developments in study design, funding mechanisms, and the reimbursement landscape. Together, these developments may help mitigate common barriers to rigorous evaluation of mobile health apps and allow society to understand whether they live up to their transformative potential.

Perceived Barriers To Rigorous Evaluation Of Mobile Health Apps—And How To Overcome Them

Barrier 1: Rigorous Evaluation Takes Too Long

One major issue preventing rigorous evaluation of mobile health apps is the cultural mismatch between mobile app developers and health intervention evaluators. App developers expect the fast pace and rapid pivots of Silicon Valley where knowledge is derived from constant streams of data and A/B testing, “moving fast and breaking things.” Evaluators of health interventions are more entrenched in the culture of academic research, where evidence takes several years to generate and release to the public. Over the time it takes to design a rigorous evaluation, find funding, get ethics approval, recruit participants, conduct a trial, and publish results—typically at least five years—an evaluated mobile health app could have changed dramatically, or even become obsolete.

New developments in trial design may help overcome this barrier. For example, in a “micro-randomized trial,” participants are not randomized to use or not use the “complete” version of an app. Instead, each individual participant is randomized many times over the course of the trial (for example, daily or multiple times per day) with respect to timing of notifications (for example, motivational messages for exercise or instructions to engage in a specific behavior). Micro-randomized trials may require fewer participants than a traditional randomized controlled trial and therefore may be completed more quickly while retaining adequate statistical power. Thus, micro-randomized trials identify the “active ingredients” in the app underlying its effectiveness that should be retained in the final design.

Barrier 2: Rigorous Evaluation Is An Unnecessary Business Risk

Another issue preventing rigorous evaluation of mobile health apps is that developers may benefit from uncertainty about whether their product works. If developers can achieve their objectives without a rigorous evaluation, why take the risk of doing a study that might show no effect? Ambiguity could be better than a negative result, especially for companies without a deep product pipeline.

Traditional randomized controlled trials can also be used to compare app elements and a well-designed evaluation can mitigate some of this risk by selecting specific elements of the app to evaluate. For example, one of us (EB) is collaborating with WayBetter on a randomized controlled trial of QuitBet, a cigarette smoking cessation mobile app. QuitBet includes elements of financial incentives and social gaming to motivate users to quit smoking. Since the effectiveness of financial incentives (especially the deposit contract incentive structure used in QuitBet) for smoking cessation is already well-established, the trial is designed to evaluate the contribution of the app’s unique social gaming elements. These elements include additional education information provided by a game “host” as well as social interaction and competition among players. All trial participants receive the financial elements, but participants are randomized to receive or not receive the social gaming elements.

Barrier 3: Rigorous Evaluation Is Too Hard

Another barrier to rigorous evaluation is the difficulty of designing a trial that stands up to scientific scrutiny. The conflicts of interest inherent in industry research necessitate engagement of an independent evaluation partner that will commit to publishing the results, positive or negative, in a trusted, peer-reviewed venue. The complexity of engaging an independent evaluation partner may not seem worthwhile to mobile app developers accustomed to conducting their own analyses.

Engaging an independent evaluation expert will provide additional value beyond credibility. Good evaluation partners will also help mobile app developers avoid common pitfalls in trial design. One such pitfall is a poorly designed control condition. Good evaluation partners will consider what a relevant comparator would be and how to define it. Without a proper control group, issues of regression to the mean and selection bias render results difficult to interpret.

Good evaluation partners will also help with other critical areas of research design, including the appropriate user subpopulation. They will make design choices around trial length and sample size that will reveal statistically or clinically important differences if they exist. Furthermore, a good evaluation partner will evaluate appropriate outcomes, including risks as well as benefits.

Barrier 4: Rigorous Evaluation Costs Too Much

For many mobile app developers, cost is a key barrier to conducting a rigorous evaluation. A small start-up may not have the resources to engage an evaluation partner. Even developers with more resources may not want to commit their own resources for a rigorous evaluation.

Mobile apps with significant potential to improve human health may be able to overcome this barrier by obtaining external funding. For example, the National Institutes of Health Small Business Innovation Research/Small Business Technology Transfer (SBIR/STTR) grant program aims to help businesses successfully commercialize their product following this period of research and development. Current SBIR/STTR funding priorities emphasize “developing mobile health technologies driven by clinical needs and integrating these technologies in healthcare delivery, wellness, and daily living.”

The funding available is substantial, and eligibility criteria are broad. Businesses based in the US and with 500 or fewer employees may be eligible to apply for funding in two phases. Feasibility and proof-of-concept studies are completed in Phase I, typically over six months with an approximately $250,000 budget (33 percent may be subcontracted). Promising Phase I studies may continue development in Phase II. Phase II may last two years with a $1.7 million budget (50 percent may be subcontracted). Developers who have accomplished the objectives expected in a Phase I grant through alternative funding mechanisms may apply directly for Phase II funding.

Policy Levers For Promoting The Rigorous Evaluation Of Mobile Health Apps

Given the relatively low physical and psychological risks to patients of using most mobile health apps (especially compared to pharmaceutical drugs or medical devices), it is unlikely that the FDA will ever mandate their rigorous evaluation. Nonetheless, mobile health apps still have the potential to cause harm to patients. This harm could be the opportunity costs of time and resources spent on ineffective apps. It could be the harm of avoiding an existing effective treatment in favor of using an app that is later found to be ineffective. Furthermore, mobile health apps can even cause harm if they contain factually incorrect information or promote activities that are healthy for most patients but harmful to patients with certain characteristics. Here are some steps that corporate and nonprofit organizations could take in the absence of a government mandate to strengthen incentives for rigorous evaluations of mobile health apps for the benefit of society.

Policy Lever 1: Independent Review And Approval of Apps By Trusted Groups

Clinicians are becoming more willing to recommend apps to their patients, but it is impossible for anyone to keep up with the volume of available options. To help clinicians make evidence-based recommendations to their patients, professional clinician societies or other nonprofit foundations could create a voluntary set of standards of evidence that mobile health apps would have to meet for endorsement. These standards could be an extension of existing app evaluation initiatives, such as MIND, the m-Health Index & Navigation Database. However, in initiatives such as these, rigorous evaluation plays only a small role currently. In MIND, having “supporting studies” is only one of 88 criteria on which apps are rated, and only a small number of apps fulfill this criterion.

Policy Lever 2: Health Care Industry Organizations Adopt And Publicize Standards Of Evidence

Health care industry players have started contracting with mobile health app developers to provide free or discounted access to the people they serve. For many developers, such partnerships with health insurers, health systems, or other organizations could represent the path to long-term financial sustainability. Thus, large health care industry organizations have substantial influence over the standards that mobile health apps would have to meet to be considered for contracting and reimbursement. These organizations could adopt existing standards, such as those endorsed by trusted professional organizations such as the Institute for Healthcare Improvement, or create their own. They would then publicize their standards for consideration for reimbursement, which would give mobile app developers a minimum standard to meet.

The Business Case For Rigorous Evaluation Of Mobile Health Apps

In the decade since the National Institutes of Health mHealth Evidence Workshop highlighted challenges to rigorous evaluation of mobile apps, there have been efforts to bring more rigor to mHealth evaluation. Overall, however, uptake of these efforts has been slow. Barriers to rigorous evaluation remain, but advancements in trial design and funding mechanisms can help overcome them. Trusted professional or nonprofit organizations and health care industry players could accelerate this process by adopting and publicizing standards of evidence required for endorsement or consideration of reimbursement.

Health behaviors are hard to change. Patients want help that is convenient and effective. Mobile health apps have the potential to transform human health by helping patients make better decisions, forge new habits, and incorporate self-care practices into their daily lives. We need better evidence that mobile health apps will do what they say they do, or we risk spending our time and money on something that may not help at all. Closer integration with health care delivery systems will only make the business case for rigorous evaluation of health mobile apps more compelling.

Authors’ Note

Erika Litvin Bloom was previously a consultant for WayBetter, Inc. She is currently the co-principal investigator of a National Institute on Drug Abuse-funded Small Business Innovation Research (SBIR) grant awarded to WayBetter, Inc.