Evaluation criteria for health apps supporting medication adherence in early-stage technology development

000317 10.3205/000317 urn:nbn:de:0183-0003171 Review Article Evaluation criteria for health apps supporting medication adherence in early-stage technology development – a scoping review Bewertungskriterien für Gesundheits-Apps zur Unterstützung der Medikamenteneinnahme in frühen Entwicklungsphasen der Technologie – ein Scoping Review Niemann Niemann Anja A

Institute for Health Care Management and Research, University Duisburg-Essen, Essen, Germany

author Hüer Hüer Theresa T

Institute for Health Care Management and Research, University Duisburg-Essen, Essen, Germany

author Neumann Neumann Anja A

Institute for Health Care Management and Research, University Duisburg-Essen, Essen, Germany

author Wasem Wasem Jürgen J

Institute for Health Care Management and Research, University Duisburg-Essen, Essen, Germany

author Schnell-Inderst Schnell-Inderst Petra P

Institute of Public Health, Medical Decision Making and Health Technology Assessment, Department of Public Health, Health Services Research and Health Technology Assessment, UMIT TIROL – University for Health Sciences and Technology, Hall i.T., Austria

author Neusser Neusser Silke S Dr.

Institute for Health Care Management and Research, University Duisburg-Essen, Thea-Leymann-Str. 9, 45127 Essen, GermanyInstitute for Health Care Management and Research, University Duisburg-Essen, Essen, Germany

silke.neusser@medman.uni-due.de author German Medical Science GMS Publishing House

Düsseldorf

610 mobile application mHealth health app drug therapy systematic review assessment criteria mobile Anwendung mHealth Gesundheits-App Arzneimitteltherapie systematische Übersicht Bewertung Kriterien Health Technology Assessment 20220428 20230404 engl This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung). 1612-3174 21 GMS German Medical Science GMS Ger Med Sci 03 German Institute of Medical Documentation (DIMDI) Einführung: Gesundheits-Apps bieten einen möglichen Lösungsansatz zur Verbesserung des Medikamentenmanagements durch Patienten. Zwar wurde mit dem Digitale-Versorgung-Gesetz (DVG) ein Anspruch in der Gesetzlichen Krankenversicherung (GKV) geschaffen, aber die Vielzahl angebotener Gesundheits-Apps mit schwankender Qualität machen es dem Leistungserbringer und vor allem dem medizinischen Laien schwer, eine bedarfsgerechte und qualitativ hochwertige Medikations-App auszuwählen. Auch Hersteller benötigen für die Entwicklung von qualitativ hochwertigen Apps von Anfang an Orientierungshilfen. Bislang liegen diverse allgemeine Bewertungskonzepte für Gesundheits-Apps vor. Welche Anforderungen Gesundheits-Apps erfüllen sollten, hängt jedoch maßgeblich von der Einsatzsituation beziehungsweise dem Typ der App ab. Daher ist die Zielsetzung des vorliegenden Beitrags, einen Überblick über die internationale Evidenzlage zu spezifischen Kriterien zur Bewertung von Medikations-Apps zu schaffen.Methoden: Im Rahmen eines Scoping Reviews wurde eine systematische Recherche am 29. Januar 2020 in PubMed und EMBASE durchgeführt. Die Suche wurde auf Publikationen ab dem Jahr 2007 und auf englische sowie deutsche Artikel beschränkt. Ergänzt wurde diese durch eine teilsystematische Recherche in den Referenzlisten der zuvor eingeschlossenen Artikel sowie auf Internetseiten von relevanten Stakeholdern. Einschlusskriterien waren: Die Publikation befasst sich mit Gesundheits-Apps, die auf dem Smartphone genutzt werden können und ihren Fokus auf die Unterstützung der Medikamenteneinnahme legen. Zudem durfte die Publikation nicht ausschließlich auf Bewertungskriterien für eine einzelne App bezogen sein. Die identifizierten Publikationen wurden in einer qualitativen Inhaltsanalyse nach Bewertungskriterien untersucht und diese in Anlehnung an die Rahmenkriterien des DVG bzw. der Digitale-Gesundheitsanwendungen-Verordnung (DiGAV) kategorisiert.Ergebnisse: Die systematische Recherche ergab 2.542 Treffer (999 in PubMed, 1.543 in EMBASE, 560 Dubletten). Insgesamt erfüllten 16 S</PlainText></TextGroup>tudien die Einschlusskriterien. Die teilsystematische Recherche identifizierte eine weitere Studie. Der auf Basis der 17 Studien entwickelte Kriterienkatalog umfasst die Oberthemen „Patientenorientierung“ (Datenschutz und -sicherheit, Verbraucherschutz, Nutzerfreundlichkeit) und „Qualität/Kernfunktionen von Medikations-Apps“ (Erinnerung, Selbstüberwachung, (Arzneimittel-)Informationen, Motivation zur Verhaltensänderung, Arzneimittel-/Patientensicherheit, Robustheit) sowie den Bereich „Interoperabilität/Zusammenarbeit“. Aufgrund ihrer spezifischen Bedeutung für Medikations-Apps sticht unter dem Oberthema „Qualität/Kernfunktionen der Medikations-Apps“ die Unterkategorie „Motivation zur Verhaltensänderung“ besonders hervor. Diese zielt auf die Bewertung der Ausgestaltung einzelner Funktionen hinsichtlich ihres Potenzials, das Verhalten der App-Nutzer tatsächlich zu verändern, ab.</Pgraph><Pgraph><Mark1>Diskussion:</Mark1> Die in der DiGAV genannten Kriterien zur Bewertung von Gesundheits-Apps wiesen eine große Schnittmenge mit den in der Literaturrecherche identifizierten Kriterien auf. Allerdings wurde der Bereich der positiven Versorgungseffekte mit den eingeschlossenen Studien kaum erfasst. Im Rahmen der Entwicklung des Kriterienkatalogs konnte keine Gewichtung der identifizierten Kriterien vorgenommen werden. Daher sollte der Katalog von Leistungserbringern, Herstellern und/oder Nutzern als eine unterstützende Checkliste verstanden werden.</Pgraph><Pgraph><Mark1>Fazit:</Mark1> Es konnte eine große Vielfalt möglicher Bewertungskriterien für Medikations-Apps gezeigt werden. Künftige Forschung sollte den Fokus auf Möglichkeiten der Gewichtung dieser vielfältigen Bewertungskriterien legen und dafür neben klinischen Studien auch Methoden der Präferenzerhebung einsetzen.</Pgraph></Abstract> <Abstract language="en" linked="yes"><Pgraph><Mark1>Introduction:</Mark1> Health apps offer an approach to improve the patients’ management of their medication. Although the Digital Healthcare Act (DVG) has created a claim in the statutory health insurance (SHI), the large number of health apps available and their varying quality make it difficult for service providers and especially for medical laypersons to select an adequate high-quality medication app. Manufacturers need guidance for the development of high-quality apps right from the start. Various general evaluation concepts for health apps have been available to date. However, the requirements that should be met by healthcare depend largely on the field of application and the type of apps. This article aims to provide an overview of the international evidence on specific criteria for the evaluation of medication apps.</Pgraph><Pgraph><Mark1>Methods:</Mark1> Within the framework of a scoping review, a systematic search was conducted in PubMed and EMBASE on January 29, 2020. The search was limited to publications from 2007 onwards as well as to English and German articles. Additionally, a semi-systematic research of reference lists of the previously included articles as well as a structured search of websites of relevant stakeholders were conducted. Inclusion criteria were the following: the publication deals with health apps that can be used on smartphones and focus on supporting medication intake; the publication does not refer to evaluation criteria for a single app exclusively. The included publications were examined in a qualitative content analysis searching for evaluation criteria and categorizing them according to the framework criteria of the DVG and the Digital Health Applications Ordinance (DiGAV).</Pgraph><Pgraph><Mark1>Results:</Mark1> 2,542 articles were identified in the systematic search (99<TextGroup><PlainText>9 i</PlainText></TextGroup>n PubMed, 1,543 in EMBASE, 560 duplicates). A total of 16 studies met the inclusion criteria. The semi-systematic research and the structured search identified one further study. A catalog of criteria was developed based on the included 17 studies. This catalog covers the general topics “patient orientation” (data protection and security, consumer protection, user friendliness) and “quality/core functions of medication apps” (reminder, self-monitoring, (drug) information, motivation to change behavior, drug/patient safety, robustness) as well as “interoperability/cooperation”. Due to its specific importance for medication apps, the subcategory “motivation for behavioral change” stands out beneath the general topic “quality/core functions of medication apps”. This category aims to evaluate the design of individual functions with regard to their potential to actually change the behavior of app users.</Pgraph><Pgraph><Mark1>Discussion:</Mark1> The criteria for the evaluation of health apps mentioned in the DiGAV intersected with the criteria identified in the literature research. However, the area of positive health care effects was hardly covered by the included studies. In the development of the criteria catalog, it was not possible to weight the identified criteria. Therefore, the catalog should be understood as a supporting checklist for service providers, manufacturers, and/or users.</Pgraph><Pgraph><Mark1>Conclusions:</Mark1> A large variety of possible evaluation criteria for medication apps could be shown. Future research should focus on the possibilities of weighting these diverse evaluation criteria, using not only clinical studies but also methods to identify preferences.</Pgraph></Abstract> <TextBlock linked="yes" name="Background"> <MainHeadline>Background</MainHeadline><Pgraph>Insufficient adherence is a widespread problem in drug therapy. The WHO defines adherence as follows: “[Adherence is] the extent to which a person’s behavior – taking medication, following a diet, and/or executing lifestyle changes, corresponds with agreed recommendations from a health care provider” <TextLink reference="1"></TextLink>. Studies reveal different, but alarming results. These depend on the indication and patient population considered, as well as on the definition used (exclusively non-adherence or additionally partial adherence). The World Health Organization (WHO) <TextLink reference="1"></TextLink> reports that about 50% of medications are not taken as prescribed. Schäfer <TextLink reference="2"></TextLink> refers to studies according to which about 20% of all patients do not redeem their prescriptions at all, and half of the remaining 80% of prescriptions are not used at all or not used as prescribed. The consequences of inadequate adherence are complex and affect the health of the patient through medical complications, treatment failure and restrictions on quality of life, but also health care costs and the national economy as a whole. These economic consequences mainly result from increased hospitalization and nursing home admissions, additional costs for therapeutic measures or drugs, and work absenteeism <TextLink reference="2"></TextLink>. For example, one in four hospital admissions is reported to be directly or indirectly related to incorrect medication use <TextLink reference="2"></TextLink>.</Pgraph><Pgraph>The WHO <TextLink reference="1"></TextLink> describes forgetting to take the correct dosage as the most common reason for non-adherence. However, numerous other factors also play a role in the correct intake of medications. These include the timing of intake or even foods that are ingested in combination and could trigger interactions. In addition, with the aging population, the proportion of multimorbid individuals taking multiple medications simultaneously is also growing <TextLink reference="3"></TextLink>. For example, a U.S. study identifies polypharmacy in 39% for the age group over 65 years. According to the most commonly used definition, polypharmacy is defined as the daily use of five or more medications <TextLink reference="4"></TextLink>. However, a significant increase in polypharmacy can also be observed in persons 20 to 39 years old <TextLink reference="4"></TextLink>. Adherence in drug therapy can thus become a complex challenge for the individual patient.</Pgraph><Pgraph>One possible approach to improving medication management by patients is offered by health apps to support medication intake (hereinafter medication apps). The smartphone is ubiquitous among many people and can thus be reliably integrated into everyday life. Apps for documenting and reminding patients to take their medications are used increasingly. According to a 2017 survey of 18- to 64-year-olds in Germany conducted by the opinion research company Statista <TextLink reference="5"></TextLink>, 12% have used a health app within the last twelve months. Specific medication apps offer features such as medication reminders, tracking of doses taken or skipped, an overview of current medication, warnings of drug interactions, medication refill reminders, medication information, and sharing of medication data with third parties <TextLink reference="6"></TextLink>. There is evidence that medication apps can improve medication adherence among users. A systematic review included a total of 17 comparative studies examining the effect of medication apps on medication adherence. In twelve of these 17 studies, a significant improvement in adherence was shown for at least one adherence measurement method <TextLink reference="7"></TextLink>. With the entry into force of the Digital Health Care Act (DVG) <TextLink reference="8"></TextLink> on December 19, 2019, the entitlement of insured persons in the Statutory Health Insurance (SHI) to health apps was anchored in the German Social Code (SGB) V. The health apps must either be prescribed by a contract doctor or psychotherapist or be approved by the health insurer (§§ 33a, 139e SGB V). Due to the high attractiveness of the healthcare market and the reimbursement option recently created by the DVG in the SHI system, numerous providers of health apps can be expected to enter the SHI market. A large number of health apps are already available in the app stores, and this number is expected to grow. The quality of these apps varies <TextLink reference="9"></TextLink>. This confusing market already makes it difficult for the medical layperson and will also confront the contract physician with the challenge of selecting a suitable application for his or her needs or those of the app user <TextLink reference="9"></TextLink>. The DVG addresses this issue by limiting the claim to health apps that will be included in a directory to be maintained by the Federal Institute for Drugs and Medical Devices (BfArM) after comprehensive review. The Digital Health Applications Ordinance (DiGAV) <TextLink reference="10"></TextLink> to the DVG, which entered into force on April 21, 2020, specifies the requirements for digital health apps to be included in the directory according to Section 139e SGB V. A guideline on this procedure prepared by the BfArM <TextLink reference="11"></TextLink> includes precise explanations for manufacturers, service providers and users.</Pgraph><Pgraph>Manufacturers need guidance for the development of high-quality apps right from the start <TextLink reference="12"></TextLink>. Currently, there is a variety of general evaluation criteria and concepts that manufacturers can use. For example, in October 2019, the Quality Criteria Core Set for Digital Health Apps (DiGA) “AppQ” was published by the Bertelsmann Foundation <TextLink reference="13"></TextLink> with the participation of Fraunhofer FOKUS. The development was funded by the German Federal Ministry of Health. This core set consists of <TextGroup><PlainText>24 c</PlainText></TextGroup>riteria organized into the nine topics of <Mark2>medical quality, positive care effects, data protection, information security, technical quality, consumer protection and fairness, interoperability, user friendliness and motivation, as well as the connection to the healthcare system</Mark2>. The criteria are applied via a web application to collect self-reported data from providers of DiGA certified as a medical device. The starting point for the development of the “AppQ” core set of quality criteria was the “APPKRI” meta-criteria catalog for the description and evaluation of health apps previously developed by Fraunhofer FOKUS <TextLink reference="14"></TextLink>, the development of which was also funded by the German Federal Ministry of Health. In addition, there are various international efforts to create transparency in the field of health apps. In some countries, for example, public health institutions offer platforms with a selection of quality-checked health apps. One example is the NHS Digital Apps Library of the National Health Service in the United Kingdom <TextLink reference="15"></TextLink>, which was published in 2017 and provides users and care providers with a selection of trusted health apps on different topics. A five-step review process assesses requirements in the nine areas of clinical effectiveness (e.g., evidence-based, clarity of purpose and use of the app), regulatory compliance (e.g., declaration as a medical device), clinical safety (e.g., potential adverse effects of the app), privacy and confidentiality (e.g., privacy statement, consent), (IT) security (e.g., data storage, network communication), usability and accessibility, interoperability (e.g., compatibility with other systems), technical stability (e.g., defined plan for product development), and change management (e.g., version control).</Pgraph><Pgraph>However, the requirements that health apps should meet depend largely on the objective of the app (for example, diagnostic support, companionship for various somatic and psychiatric conditions, documentation of health data, support for a healthy lifestyle). For example, medication apps with the goal of promoting adherence focus on changing patient behavior. In order to create a meaningful evaluation basis (in terms of criteria and instruments) for medication apps, systematic as well as supplementary semi-systematic research on the international evidence base has been conducted as part of a scoping review.</Pgraph></TextBlock> <TextBlock linked="yes" name="Research questions"> <MainHeadline>Research questions</MainHeadline><Pgraph>The aim of the present study is to examine the extent to which the legal requirements in Germany are consistent with the results on evaluation criteria retrieved from international literature. However, the evaluation criteria to be identified for medication apps should not be considered in isolation, but should always be classified against the background of the interests of users, manufacturers, and healthcare providers. Finally, the extent to which the identified criteria cover relevant areas of the DiGAV will be considered.</Pgraph></TextBlock> <TextBlock linked="yes" name="Methodology"> <MainHeadline>Methodology</MainHeadline><Pgraph>This article was based on a two-stage approach. In the first phase, studies were identified that addressed evaluation criteria for medication apps. The second phase involved the development of a criteria catalog and the assignment of the previously identified criteria to it.</Pgraph><Pgraph>In the first phase, studies were sought that focused on the criteria themselves. For this purpose, the scoping review method was used, which aims to provide an overview of the currently available evidence in the literature, without assessment of the quality of the publications or restrictions regarding the type of study. It can also identify gaps in the evidence <TextLink reference="16"></TextLink>. First, a systematic literature search was performed in the PubMed and E<TextGroup><PlainText>MB</PlainText></TextGroup>ASE databases on January 29, 2020. Search terms on the top-level topics of health apps, medication adherence, and evaluation criteria were linked using the Boolean operator AND. In addition, the search was limited to results from 2007 onwards, the release year of the first iPhone <TextLink reference="17"></TextLink>, and to English and German articles. Conference transcripts, letters, and notes were excluded. In the next step, duplicates were removed. Subsequent screening of titles and abstracts, as well as subsequent full-text screening was performed independently by two persons. Discrepancies were resolved by discussion. Publications were included if the following criteria were met: the publication addresses health apps that can be used on smartphones and focus on medication adherence support. In addition, the publication had to consider evaluation criteria and tools for medication apps in general and could not be exclusively related to a specific app. Publications were excluded based on the following criteria: abstract or full text were not available; the health apps studied were primarily used for diagnosis, research, or information, aimed at healthcare providers, limited to SMS exchange, or did not represent standalone softwa<TextGroup><PlainText>re. T</PlainText></TextGroup>he search terms are detailed in Appendix 1 (Attachment 1 <AttachmentLink attachmentNo="1"/>) and the inclusion and exclusion criteria are detailed in Appendix 2 (Attachment 1 <AttachmentLink attachmentNo="1"/>). In addition, a supplementary semi-systematic search was conducted as part of the scoping review. Reference lists of articles included in the systematic search were screened for relevant publications. In addition, websites of stakeholders, associations and public institutions were searched for relevant publications (see Attachment 1 <AttachmentLink attachmentNo="1"/>, Appendix 3).</Pgraph><Pgraph>The starting point for the second phase was the evaluation criteria for medication apps identified in the first phase, which until then still existed in a loose unsystematic collection. The central added value of this study lies in the development of a catalog of criteria and the assignment of the previously identified individual criteria to this catalog. The starting point for this was the analysis of the DVG <TextLink reference="8"></TextLink>. This defines the requirements for digital health applications, which also include health apps, for inclusion in the directory maintained by the BfArM in accordance with § 139e SGB V. These include requirements for safety, functional suitability, quality, data protection and data security, but also the existence of positive effects on care. The details of these requirements are specified in the DiGAV <TextLink reference="10"></TextLink>. Two annexes could be taken from this regulation, in which very detailed question catalogs specify the requirements or criteria. These legally defined criteria for the evaluation of health apps were incorporated into the structure of the criteria catalog. The structure derived from the law was adapted and supplemented against the background of the specific medication reference of this study. Only categories for which evaluation criteria could be identified in the literature search were included in the criteria catalog. Categories that were omitted from the criteria catalog with respect to the legally derived structures are explained in more detail in the discussion section. The additions to the structure of the criteria catalog to the categories derived by law resulted from conspicuously frequently mentioned criteria of the included studies that could not be classified in the legal grid. The final criteria catalog was consented by two authors of this paper. In a next step, the included studies were analyzed within the framework of a coding system, and their individual criteria were assigned to the developed criteria catalog. The detailed procedure can be found in Appendix 4 (Attachment 2 <AttachmentLink attachmentNo="2"/>). This linkage of current legislation with the criteria and categories identified in the scientific literature ascribes relevance to health policy.</Pgraph></TextBlock> <TextBlock linked="yes" name="Results"> <MainHeadline>Results</MainHeadline><Pgraph>As part of the scoping review, the systematic search identified 999 hits in PubMed and 1,543 hits in EMBASE. After exclusion of 560 duplicates, 1,982 publications entered the title and abstract screening. On this basis, 1,956 publications were excluded because they either did not meet the predefined inclusion criteria and/or me<TextGroup><PlainText>t a</PlainText></TextGroup>t least one of the exclusion criteria. The inclusion and exclusion criteria can be found in Appendix 2 (Attachment 1 <AttachmentLink attachmentNo="1"/>) or, in abbreviated form, in the flow chart below (Figure 1 <ImgLink imgNo="1" imgType="figure"/>). Thus, a total of 26 studies were subjected to full text screening, in which a further ten studies were excluded. One additional study was identified via the supplementary semi-structured search. Thus, a total of 17 studies were included in the content analysis. Since the aim of this paper is to provide an overview of the specific criteria for the evaluation of medication apps that can be taken from the international evidence base, the general evaluation concepts cited in the background which have no direct reference to medication apps are not included in the analysis. However, the general assessment tools used in the included studies (e.g. MARS) are included in the analysis because they are specifically used to assess medication apps in the context of the respective studies. These assessment tools are explained in more detail below.</Pgraph><Pgraph>An overview of the 17 included studies is presented in Table 1 <ImgLink imgNo="1" imgType="table"/>. In addition to the authors, the year of publication and the country, this overview also shows the methodological approach of the studies and the instruments developed and/or tested for the evaluation of medication apps.</Pgraph><Pgraph>These are predominantly very recent studies (only 2/1<TextGroup><PlainText>7 s</PlainText></TextGroup>tudies were published before 2016) conducted in the USA (n=6), Canada (n=6), Singapore (n=2), China (n=1), Ireland (n=1), and Australia (n=1).</Pgraph><Pgraph>The included studies differ in terms of their methodology with respect to three phases of the development of an assessment tool (see also Table 1 <ImgLink imgNo="1" imgType="table"/> for details). In the first phase, criteria for the evaluation of medication apps are identified. A systematic literature review was conducted by two of the included 17 studies <TextLink reference="18"></TextLink>, <TextLink reference="19"></TextLink>. In seven studies the criteria were retrieved from a consensus of the authors and/or semi-systematic searches <TextLink reference="6"></TextLink>, <TextLink reference="20"></TextLink>, <TextLink reference="21"></TextLink>, <TextLink reference="22"></TextLink>, <TextLink reference="23"></TextLink>, <TextLink reference="24"></TextLink>, <TextLink reference="25"></TextLink>. Seven other studies adopted previously developed criteria and instruments <TextLink reference="26"></TextLink>, <TextLink reference="27"></TextLink>, <TextLink reference="28"></TextLink>, <TextLink reference="29"></TextLink>, <TextLink reference="30"></TextLink>, <TextLink reference="31"></TextLink>, <TextLink reference="32"></TextLink>. One study did not provide information on the development of the criteria list <TextLink reference="33"></TextLink>. In the second phase, an assessment instrument is developed based on the identified criteria. In the third phase, the assessment tool developed in each study was applied to evaluate medication apps. Between four <TextLink reference="26"></TextLink> and 645 <TextLink reference="28"></TextLink> medication apps were included in the different studies (see Table 1 <ImgLink imgNo="1" imgType="table"/> for more details). The evaluation of the respective selected apps was partly based only on the app descriptions and screenshots of these and/or by downloading and testing the apps. The evaluation of the medication apps was carried out by the authors of the respective study or medical specialists (n=17) and, in some cases, by the target group itself (n=2) <TextLink reference="26"></TextLink>, <TextLink reference="29"></TextLink>.</Pgraph><Pgraph>The last two columns in Table 1 <ImgLink imgNo="1" imgType="table"/> describe the evaluation instruments developed for the assessment of apps to support medication adherence. These evaluation instruments were either developed independently or adopted from other studies. Through these instruments, the criteria are measured or operationalized in the included studies.</Pgraph><Pgraph>A total of seven of the 17 included studies developed new instruments for the assessment <TextLink reference="6"></TextLink>, <TextLink reference="18"></TextLink>, <TextLink reference="20"></TextLink>, <TextLink reference="33"></TextLink>, <TextLink reference="21"></TextLink>, <TextLink reference="19"></TextLink>, <TextLink reference="25"></TextLink>. In the other studies previously developed instruments were used exclusively (n=7) <TextLink reference="26"></TextLink>, <TextLink reference="27"></TextLink>, <TextLink reference="28"></TextLink>, <TextLink reference="29"></TextLink>, <TextLink reference="30"></TextLink>, <TextLink reference="31"></TextLink>, <TextLink reference="32"></TextLink> or in addition to self-developed instruments (n=3) <TextLink reference="22"></TextLink>, <TextLink reference="23"></TextLink>, <TextLink reference="24"></TextLink>. The simplest assessment instrument is the nominal recording of a criteria. Accordingly, only the presence of certain functions or characteristics is recorded and coded binarily (yes/no or 0=criterion not present/1=criterion present). This operationalization was chosen by five of the ten studies with own contributions in the development of the assessment instrument <TextLink reference="6"></TextLink>, <TextLink reference="33"></TextLink>, <TextLink reference="23"></TextLink>, <TextLink reference="24"></TextLink>, <TextLink reference="25"></TextLink>. The remaining five studies with self-developed instruments operationalized their criteria with varying differentiations. In these studies, individual criteria were weighted. For example, a subsequen<TextGroup><PlainText>t w</PlainText></TextGroup>eighting in a dichotomous response format was applied (n=3) <TextLink reference="20"></TextLink>, <TextLink reference="21"></TextLink>, <TextLink reference="22"></TextLink>. In addition, weighting was done with response options to which fixed points are assigned (n=2) <TextLink reference="19"></TextLink>, <TextLink reference="24"></TextLink> and/or Likert scales, i.e., a multilevel response scale with more or less strong agreement or disagreement were used <TextLink reference="21"></TextLink>. In addition, a visual rating with one to five stars is used in one study <TextLink reference="21"></TextLink>. Often, however, the evaluation instruments consist of a combination of different response formats.</Pgraph><Pgraph>In addition to these newly designed evaluation instruments, established evaluation instruments were used, which were not specifically designed for the evaluation of medication apps. Although these general assessment catalogs are not directly aimed at answering the research question on medication-specific criteria, the MARS instrument, published in 2015, will be briefly discussed below due to the widespread use. It is also available in a version for end users (uMARS). MARS comprises a total of <TextGroup><PlainText>29 q</PlainText></TextGroup>uestions in the five areas engagement, functionality, aesthetics, information quality, and subjective quality scale. The questions within the first four areas are measured on a five-point Likert scale (from “inadequate” to “excellent”). The subjective quality assessment is based on predetermined answering options. A mean value is calculated for each area, which is included into the overall evaluation of all mean values <TextLink reference="34"></TextLink>, <TextLink reference="35"></TextLink>. In the studies by Carmody et al. <TextLink reference="26"></TextLink> and Morrissey et al. <TextLink reference="27"></TextLink>, apps are assessed by using a Behavior Change Technique (BCT) catalog <TextLink reference="36"></TextLink>, <TextLink reference="37"></TextLink>. Interventions that aim to change behavior are complex. The challenge is to identify the effective components within the intervention. A BCT describes a component of an intervention that is intended to change behavior, such as reinforcement or feedback. To create BCT catalogs, intervention descriptions from studies that identified techniques associated with effective outcome were used <TextLink reference="36"></TextLink>. Medication apps aim to change the user’s behavior to promote adherence. If the concept of the respective app is divided into different components, e.g., reminder function, reward system, or medication list, it is possible to identify the effective components of the apps aiming at behavior change by coding the components according to BCT catalogs. This allows for the differentiation of apps. The presence or absence (0/1) of 26 BCT items is coded in Abraham et al. <TextLink reference="37"></TextLink> and of 93 BCT items in Michie et al. <TextLink reference="36"></TextLink>. For example, Carmody et al. <TextLink reference="26"></TextLink> identify techniques in the apps studied such as detailed planning/definition of behavior (setting alarms, defining medication schedule), identification of environmental cues that remind of desired behavior (setting alarms for a specific time), or recording behavior (recording whether medication dose was taken). In addition to the use of known BCT catalogs, the other included studies (8/17) also recorded elements designed to motivate the user to change behavior <TextLink reference="6"></TextLink>, <TextLink reference="26"></TextLink>, <TextLink reference="21"></TextLink>, <TextLink reference="30"></TextLink>, <TextLink reference="22"></TextLink>, <TextLink reference="31"></TextLink>, <TextLink reference="23"></TextLink>, <TextLink reference="25"></TextLink>. These include, for example, the motivational functions or features “protocol”, “incentives”, and “interaction”. Even though the authors of these studies do not explicitly refer to these evaluation criteria as BCT, they also pursue the objective of identifying components of the tested apps that can change the user’s behavior by evaluating them based on motivational functions and features.</Pgraph><Pgraph>The criteria catalog for medication apps developed as part of this study includes the legally defined criteria and combines them with the criteria identified in the literature search. It comprises the overarching topics “patient orientation” and “quality/core functions of medication apps available” as well as the area of “interoperability/collaboration”. The categories “data protection and security”, “consumer protection”, and “user friendliness” are subsumed under the “patient orientation” area. The categories of the topic area “quality/core functions of medication apps available” include “reminder”, “self-monitoring”, “(medication) information”, “motivation for behavior change”, “medication/patient safety”, and “robustness”. In particular, the medication/adherence-specific criteria go beyond the structure of the DiGAV. The criteria identified in the literature review could first be assigned to the categories of the developed criteria catalog and then summarized as a total of 59 evaluation criteria (see also Attachment 2 <AttachmentLink attachmentNo="2"/>, Appendix 4). This assignment can be seen in Figure 2 <ImgLink imgNo="2" imgType="figure"/>. The size of the bubbles indicates the number of mentions. For example, the criterion “recording of the medication use” with 11/17 mentions is shown significantly larger in Figure 2 <ImgLink imgNo="2" imgType="figure"/> than the criterion “statistics” with 2/17 mentions.</Pgraph><Pgraph>The wide range of mentions of particular evaluation criteria in the respective studies is striking. Seven individual criteria are each listed in only one of the studies included in this scoping review. They are not shown in Figure 2, but are listed in the legend of Figure 2 <ImgLink imgNo="2" imgType="figure"/>. In contrast, the criterion “recording of the medication use” is mentioned in 11/17 publications. The category “user friendliness” shows a wide variety of evaluation aspects with eleven subcategories, whereas the categories “robustness” and “data protection and security” show a narrow range of criteria with only three criteria each. The included studies place a large focus on the area of core functions of medication apps. Among the wide range of criteria, the presence of functions for “recording of the medication use” (11/17) and “medication database” (9/17) are mentioned particularly frequently. Other frequently mentioned criteria are “data exporting/sharing” (10/17) and “collaboration with medical providers” (8/17) in the “interoperability/collaboration” category, “password protection” (8/17) in the “data protection and security” category, and “appropriateness” (8/17) in the “user friendliness” category. Furthermore, some aspects are mentioned in the studies to support the app user’s motivation to change behavior. Two BCT catalogs have been incorporated, which in turn describe a plethora of behavior change techniques as a stand-alone tool. For example, Michie et al. <TextLink reference="36"></TextLink> group 93 BCTs into the following categories: planned consequences, reward and threat, repeat and replace, environment, associations, covert learning, natural consequences, feedback and monitoring, goals and planning, social support, comparing behavior, confidence, comparing outcomes, identity, shaping knowledge, and regulation.</Pgraph></TextBlock> <TextBlock linked="yes" name="Discussion"> <MainHeadline>Discussion</MainHeadline><Pgraph>Through the conducted literature search, 59 different criteria for the evaluation of apps for medication adherence support could be identified. In the included studies, 10/17 studies developed assessment tools based on systematic literature review, author consensus, semi-systematic literature search, or in one case without specifying the basis, and/or 10/17 studies used existing generic tools (e.g., MARS or BCT). The 59 identified criteria were assigned to the categories of the criteria catalog developed in this study. This criteria catalog includes the headings “patient orientation” and “quality/core functions of medication apps available” as well as the area “interoperability/collaboration”. The three categories “data protection and security”, “consumer protection” and “user friendliness” are subsumed under the “patient orientation” area. Categories in the topic area “quality/core functions of medication apps available” include six categories of “reminder”, “self-monitoring”, “(medication) information”, “motivation to change behavior”, “medication/patient safety”, and “robustness”. While some of the categories specifically target medication apps, others are also relevant in the evaluation of general health apps.</Pgraph><Pgraph>The category “motivation to change behavior” stands out in the developed criteria catalog due to its specific importance for medication apps and its special design and will be briefly discussed below. This category aims to evaluate the design of individual functions in terms of their potential to actually change the behavior of app users. In two studies <TextLink reference="26"></TextLink>, <TextLink reference="27"></TextLink>, detailed question catalogs with evidence-based BCT were used for this purpose. In addition, individual criteria were addressed in eight of the 17 included studies in this area, such as incentives or goal setting. The two included papers using the BCT catalogs assigned the components of the tested apps to specific behavior change techniques (BCT). Via this mapping, it is possible to identify the components of an app that are effective in terms of behavior change. Assuming that motivation for behavior change is one of the main goals of medication apps, a qualitative evaluation of individual components and their specific design can be carried out that way. When designing a medication app, the manufacturer can examine components of the app concerning the criteria of motivation to change behavior and thus focus on particularly effective components and their design.</Pgraph><Pgraph>Since the DiGAV and the BfArM guideline were important starting points for the development of the criteria catalog presented in this study, a brief comparison of the central legal requirements and the included studies will be conducted below. Overall, the appendices of the DiGAV contain comprehensive questionnaires with a higher level of detail than the included studies. These questionnaires serve as a tool for part of the evaluation categories defined by the DiGAV. These include data privacy and security (Appendix 1 of the DiGAV, <TextLink reference="10"></TextLink>) as well as interoperability, robustness, consumer protection, user friendliness and accessibility, provider support, medical content quality, and patient safety (Appendix 2 of the DiGAV, <TextLink reference="10"></TextLink>). Via yes-no statements as well as the statement “not applicable” the manufacturer confirms the queried criteria. A “not applicable” answer that is not given for selection requires a written explanation of why the criterion has not been fulfilled yet <TextLink reference="10"></TextLink>. The category of positive health care effects is explained in the text of the regulation and ranks highly in the DiGAV <TextLink reference="10"></TextLink> for inclusion in the Digital Health Applications Directory <TextLink reference="10"></TextLink>. According to Section 10 of the DiGAV <TextLink reference="10"></TextLink>, <TextLink reference="11"></TextLink>, the manufacturer must demonstrate positive care effects with quantitative comparative studies. These can be clinical or epidemiological studies, but methods from other scientific fields such as health services research, social research, or behavioral research are also permissible, as long as the chosen method fits the subject of the study. Not only prospective, but also retrospective studies (including data from digital patient records, routine data, and registries) are licit. For example, case-control studies, retrospective cohort studies, or even intraindividual comparisons are applicable <TextLink reference="10"></TextLink>. In addition to the positive health care effects mentioned in the regulation, the criteria catalog developed in this paper lists individual criteria that correspond to the DiGAV <TextLink reference="10"></TextLink> categories of the coordination of treatment processes (e.g., criterion “collaboration with medical providers”), patient safety (e.g., criterion “interaction/side effects”), and health literacy (e.g., “medication information”). For the important categories of prescription adherence and medical benefit (shortening of disease duration, prolongation of survival, improvement of quality of life), however, the criteria catalog does not provide any criterion that directly measures these dimensions. Thus, in the included studies, the assessment of medical effects by criteria plays no or a rather subordinate role. This will be explained in more detail in the following sections. Furthermore, the DiGAV mentions the CE conformity marking as the only aspect for assessing safety and functional suitability. This EU marking is not addressed in the included studies, which were primarily conducted in non-EU countries; however, comparable, general markings are not mentioned either. The category of interoperability and collaboration, which is highlighted in the DiGAV, is also found in the included studies. This involves, for example, resetting an app to its initial state, plausibility checks when entering data, transferring data to the electronic patient record, or sharing data with care providers and the social environment. Overall, it is clear that the structure and content of the criteria catalog developed in this paper based on the included studies show a large overlap with the requirements of the DiGAV. Only the area of positive care effects was hardly covered by the included studies.</Pgraph><Pgraph>For quality assurance purposes, the model developed by Arah et al. <TextLink reference="38"></TextLink> in 2006 for the comparative assessment of health care systems in OECD member countries (OECD=Organisation for Economic Co-operation and Development) will be applied to the list of criteria developed in this study. In this model, criteria for assessing the performance of a health care system are grouped into specific dimensions in a condensed list. The comparison shows that the aspects of the OECD model safety of care (esp. via categories “data protection and security”, “consumer protection”, “medication/patient safety”), patient-centeredness (esp. via categories “user friendliness” and “motivation to change behavior”), access to care (esp. category “user friendliness”), and expenditures/costs (esp. criterion “free of charge” in category “consumer protection”) are addressed by the catalog of criteria developed in this study. However, the main segment of effectiveness described in the OECD model cannot be filled with criteria.</Pgraph><Pgraph>The lack of evaluation criteria regarding the effectiveness or positive effects of medication apps on care is due to the fact that the evaluation criteria in the included studies primarily focus on the range of functions and technical features of the apps. This in turn is due to the methodology of the included studies. The evaluation of the selected apps in the third phase of the included studies was partly based only on the app descriptions and screenshots of these and/or by downloading and testing the apps over a very short period of time by the authors and not by users themselves (the latter in only 2/17 studies). Hence, only dimensions that are measurable in the early development phase of the technology and thus can be assessed in the short term can be considered according to the objective of this scoping review. As to the identified criteria, it could not be determined whether and to what extent there are correlations between a particular evaluation criterion and long-term medical benefits. The evaluation of criteria concerning long-term effectiveness would have to be implemented in clinical trials of a comparative nature in which patients or app users are included. Clinical evaluation of care effects could be measured with outcome parameters related to adherence or drug safety/interactions. At the early stages of technology development, systematic reviews of clinical trials on medication apps might help to identify possibly effective app components that might contribute to the improvement of the quality of care. In addition, manufacturers could evaluate medication app components according to the BCT catalogs described earlier in order to gain an impression of the app’s potential in terms of user behavior change.</Pgraph><Pgraph>Furthermore, it should be emphasized that the frequency of criteria mentions does not yet indicate their relevance. It was not possible to weight the identified criteria as part of the development of the criteria catalog. Thus, from this scoping review app developers cannot derive which aspects picking up on the evaluation criteria should be given special emphasis in the design of the app in order to generate the greatest possible medical benefit for the app user. To weight the evaluation criteria identified in this scoping review, further research is essential, particularly in the form of comparative clinical studies that establish and quantify a relationship between the fulfillment of certain criteria and the occurrence of medical effects. This may confirm criteria that are associated with increased patient benefit in the long term. In addition, the relevance of criteria could differ greatly with respect to the perspective taken. Therefore, preference surveys, such as choice-based conjoint analyses, could be a useful addition as a basis for weighting criteria. In these, alternatives could be evaluated as a bundle of their characteristics (i.e., specific evaluation criteria) from different perspectives (service provider, user, payer) in an intuitive decision-making process. For example, from the perspective of the treating physician, criteria that aim at an efficient (data) exchange with the patient and evaluate the effectiveness of the medication app could be important. Users, on the other hand, might consider criteria that evaluate the presentation of medical content and the handling of the app to be more important. A mapping of preferences to the criteria identified in this study with regard to different target groups could improve the use as a targeted basis for decision-making.</Pgraph></TextBlock> <TextBlock linked="yes" name="Limitations"> <MainHeadline>Limitations</MainHeadline><Pgraph>In the included studies, the mentioned criteria are presented in different forms, for example as a sentence or only as a short bullet point, and different formulations for criteria are chosen. The summary of the criteria of the included studies into unified evaluation criteria can be differentiated to varying degrees. With this in mind, a balance between detailed differentiation and clear summary was sought in this study and developed by consensus as understood by the authors of this paper. This also applies to the chosen structure of the criteria catalog, which is based on the results found in the studies in addition to the structure of the DiGAV. Moreover, in the context of a scoping review, the evidence from the literature is reported regardless of the quality of the included studies, so that all criteria found were equally mentioned and summed up. Furthermore, the selected inclusion and exclusion criteria of the literature search were used to identify studies that addressed the design of evaluation criteria/instruments for medication apps in order to apply them to the evaluation of a set of different apps. Studies that only addressed the evaluation of a specific app were excluded. It is possible that evaluation criteria could have been indirectly derived from these publications as well.</Pgraph></TextBlock> <TextBlock linked="yes" name="Answering the research questions"> <MainHeadline>Answering the research questions</MainHeadline><Pgraph>The criteria for evaluating health apps identified in the DiGAV had a high degree of intersection with the criteria identified in the literature review. However, the area of positive care effects was poorly captured with the included studies. Quality assurance of the criteria catalog via the OECD model according to Arah et al. <TextLink reference="38"></TextLink> revealed that the aspects of safety of care, patient-centeredness, access to care, and expenditures/costs were covered via the developed criteria catalog. The main segment of effectiveness described in the OECD model, however, could also not be fulfilled due to the short-term dimension or the rather functional orientation of the identified evaluation criteria. In addition, the frequency of mentions of the criteria alone does not indicate the relevance of a criterion. For these reasons, it was not possible to weight the identified criteria during the development of the criteria catalog. Therefore, the criteria catalog developed should not be understood by service providers, manufacturers and/or users as a comprehensive evaluation tool with a clear result, but rather as a supporting checklist. It is thus only a first step in the development of decision support tools.</Pgraph><Pgraph>Important potentials are associated with the criteria catalog developed in this article. In order to be include<TextGroup><PlainText>d i</PlainText></TextGroup>n the list of reimbursable digital health applications maintained by the BfArM in accordance with <TextGroup><PlainText>§ 139e SGB V</PlainText></TextGroup>, health apps must fulfill the required conditions of the DiGAV <TextLink reference="10"></TextLink>, which are explained by the BfArM by means of examples in the DiGAV guide. In addition to the manufacturer’s specifications, the list also includes information on the features and performance of the digital health apps listed <TextLink reference="11"></TextLink>. If the healthcare provider or a patient himself <TextLink reference="11"></TextLink> is faced with the challenge of selecting the most suitable registered application, an evaluation that goes beyond the register and is more specific to the area of medication management plays an important role. The evaluation criteria identified in the present study can serve as a basis for a qualitative evaluation of medication apps and thus provide a supplementary decision-making aid for healthcare providers and also for potential app users for the needs-based selection of medication apps that have been included in the register of reimbursable digital health apps.</Pgraph></TextBlock> <TextBlock linked="yes" name="Conclusion"> <MainHeadline>Conclusion</MainHeadline><Pgraph>In view of the ever scarcer resources in the health care system, the efficient use of drugs is essential. Schäfer <TextLink reference="2"></TextLink> states that the most expensive drug is the one that the patient does not take. Therefore, digital approaches promoting adherence should be focused on. However, medical care providers, manufacturers and users need more specific guidance than the general concepts available to date. In the present scoping review, an overview of the international evidence base on criteria for the evaluation of medication apps could be created and presented in a structured way in a criteria catalog. In addition to the studies included in the literature search, the legal requirements in Germany (DVG, DiGAV) in particular served as the basis for the development of this criteria catalog. Future research should focus on ways to weight these criteria, using preference survey methods in addition to clinical studies. Furthermore, dimensions of effectiveness or positive effects of care should be added to the criteria catalog.</Pgraph></TextBlock> <TextBlock linked="yes" name="Abbreviations"> <MainHeadline>Abbreviations</MainHeadline><Pgraph><UnorderedList><ListItem level="1">BCT: Behavior Change Technique</ListItem><ListItem level="1">BfArM: Bundesinstitut für Arzneimittel und Medizinprodukte (Federal Institute for Drugs and Medical Devices)</ListItem><ListItem level="1">DiGA: Digitale Gesundheitsanwendung (digital health application)</ListItem><ListItem level="1">DiGAV: Digitale-Gesundheitsanwendungen-Verordnung (Digital Health Applications Ordinance)</ListItem><ListItem level="1">DVG: Digitale-Versorgung-Gesetz (Digital Healthcare Act)</ListItem><ListItem level="1">ePA: Elektronische Patientenakte (electronic patient file)</ListItem><ListItem level="1">SHI: Statutory health insurance</ListItem><ListItem level="1">MARS: Mobile App Rating Scale</ListItem><ListItem level="1">Medikations-App: Gesundheitsapps zur Unterstützung der Medikamenteneinnahme (health app to support intake of medications)</ListItem><ListItem level="1">NHS: National Health Service (Great Britain)</ListItem><ListItem level="1">OECD: Organisation für wirtschaftliche Zusammenarbeit und Entwicklung (Organisation for Economic Co-operation and Development)</ListItem><ListItem level="1">SGB: Sozialgesetzbuch (social security code)</ListItem><ListItem level="1">uMARS: User Version of the Mobile Application Rating Scale</ListItem><ListItem level="1">WHO: World Health Organization</ListItem></UnorderedList></Pgraph></TextBlock> <TextBlock linked="yes" name="Notes"> <MainHeadline>Notes</MainHeadline><SubHeadline>Funding</SubHeadline><Pgraph>The project was supported by the German Institute of Medical Documentation (DIMDI).</Pgraph><SubHeadline>Competing interests</SubHeadline><Pgraph>The authors declare that they have no competing interests.</Pgraph></TextBlock> <References linked="yes"> <Reference refNo="1"> <RefAuthor>World Health Organization</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>Adherence to long-term Therapies</RefBookTitle> <RefPage></RefPage> <RefTotal>World Health Organization. Adherence to long-term therapies. [last accessed 2020 Mar 11]. Available from: https://apps.who.int/iris/handle/10665/42682?locale-attribute=de&</RefTotal> <RefLink>https://apps.who.int/iris/handle/10665/42682?locale-attribute=de&</RefLink> </Reference> <Reference refNo="2"> <RefAuthor>Schäfer C</RefAuthor> <RefTitle></RefTitle> <RefYear>2017</RefYear> <RefBookTitle>Patientencompliance. Adhärenz als Schlüssel für den Therapieerfolg im Versorgungsalltag</RefBookTitle> <RefPage></RefPage> <RefTotal>Schäfer C. Patientencompliance. Adhärenz als Schlüssel für den Therapieerfolg im Versorgungsalltag. 2nd ed. Wiesbaden: Springer; 2017.</RefTotal> </Reference> <Reference refNo="3"> <RefAuthor>Masnoon N</RefAuthor> <RefAuthor>Shakib S</RefAuthor> <RefAuthor>Kalisch-Ellett L</RefAuthor> <RefAuthor>Caughey GE</RefAuthor> <RefTitle>What is polypharmacy? A systematic review of definitions</RefTitle> <RefYear>2017</RefYear> <RefJournal>BMC Geriatr</RefJournal> <RefPage>230</RefPage> <RefTotal>Masnoon N, Shakib S, Kalisch-Ellett L, Caughey GE. What is polypharmacy? A systematic review of definitions. BMC Geriatr. 2017 Oct;17(1):230. DOI: 10.1186/s12877-017-0621-2</RefTotal> <RefLink>https://doi.org/10.1186/s12877-017-0621-2</RefLink> </Reference> <Reference refNo="4"> <RefAuthor>Kantor ED</RefAuthor> <RefAuthor>Rehm CD</RefAuthor> <RefAuthor>Haas JS</RefAuthor> <RefAuthor>Chan AT</RefAuthor> <RefAuthor>Giovannucci EL</RefAuthor> <RefTitle>Trends in Prescription Drug Use Among Adults in the United States From 1999-2012</RefTitle> <RefYear>2015</RefYear> <RefJournal>JAMA</RefJournal> <RefPage>1818-31</RefPage> <RefTotal>Kantor ED, Rehm CD, Haas JS, Chan AT, Giovannucci EL. Trends in Prescription Drug Use Among Adults in the United States From 1999-2012. JAMA. 2015 Nov;314(17):1818-31. DOI: 10.1001/ jama.2015.13766</RefTotal> <RefLink>https://doi.org/10.1001/jama.2015.13766</RefLink> </Reference> <Reference refNo="5"> <RefAuthor>Kunst A</RefAuthor> <RefTitle>Welche dieser Gesundheits-Apps und smarten Gesundheits-Geräte (mit dem Internet vernetzt) haben Sie in den letzten 12 Monaten genutzt?</RefTitle> <RefYear>2017</RefYear> <RefBookTitle>Umfrage in Deutschland zu Health-Apps und smarten Gesundheitsgeräten 2017</RefBookTitle> <RefPage></RefPage> <RefTotal>Kunst A. Welche dieser Gesundheits-Apps und smarten Gesundheits-Geräte (mit dem Internet vernetzt) haben Sie in den letzten 12 Monaten genutzt? In: Umfrage in Deutschland zu Health-Apps und smarten Gesundheitsgeräten 2017. Statista; 2017 [last accessed 2020 Mar 11]. Available from: https://de.statista.com/prognosen/810056/umfrage-in-deutschland-zu-health-apps-und-smarten-gesundheitsgeraeten</RefTotal> <RefLink>https://de.statista.com/prognosen/810056/umfrage-in-deutschland-zu-health-apps-und-smarten-gesundheitsgeraeten</RefLink> </Reference> <Reference refNo="6"> <RefAuthor>Haase J</RefAuthor> <RefAuthor>Farris KB</RefAuthor> <RefAuthor>Dorsch MP</RefAuthor> <RefTitle>Mobile Applications to Improve Medication Adherence</RefTitle> <RefYear>2017</RefYear> <RefJournal>Telemed J E Health</RefJournal> <RefPage>75-9</RefPage> <RefTotal>Haase J, Farris KB, Dorsch MP. Mobile Applications to Improve Medication Adherence. Telemed J E Health. 2017 Feb;23(2):75-9. DOI: 10.1089/tmj.2015.0227</RefTotal> <RefLink>https://doi.org/10.1089/tmj.2015.0227</RefLink> </Reference> <Reference refNo="7"> <RefAuthor>Ng R</RefAuthor> <RefAuthor>Carter SR</RefAuthor> <RefAuthor>El-Den S</RefAuthor> <RefTitle>The impact of mobile applications on medication adherence: a systematic review</RefTitle> <RefYear>2020</RefYear> <RefJournal>Transl Behav Med</RefJournal> <RefPage>1419-35</RefPage> <RefTotal>Ng R, Carter SR, El-Den S. The impact of mobile applications on medication adherence: a systematic review. Transl Behav Med. 2020 Dec;10(6):1419-35. DOI: 10.1093/tbm/ibz125</RefTotal> <RefLink>https://doi.org/10.1093/tbm/ibz125</RefLink> </Reference> <Reference refNo="8"> <RefAuthor>Anonym</RefAuthor> <RefTitle>Gesetz für eine bessere Versorgung durch Digitalisierung und Innovation (Digitale-Versorgung-Gesetz – DVG)</RefTitle> <RefYear>2019</RefYear> <RefJournal>Bundesgesetzblatt</RefJournal> <RefPage>2562</RefPage> <RefTotal>Gesetz für eine bessere Versorgung durch Digitalisierung und Innovation (Digitale-Versorgung-Gesetz – DVG). Bundesgesetzblatt. 2019 Dec 18; 2019 Part I(49): 2562.</RefTotal> </Reference> <Reference refNo="9"> <RefAuthor>Albrecht UV</RefAuthor> <RefTitle>Orientierung für Nutzer von Gesundheits-Apps</RefTitle> <RefYear>2016</RefYear> <RefBookTitle>Chancen und Risiken von Gesundheits-Apps (CHARISMHA)</RefBookTitle> <RefPage>282-300</RefPage> <RefTotal>Albrecht UV. Orientierung für Nutzer von Gesundheits-Apps. In: Albrecht UV, editor. Chancen und Risiken von Gesundheits-Apps (CHARISMHA). Medizinische Hochschule Hannover; 2016. p. 282-300.</RefTotal> </Reference> <Reference refNo="10"> <RefAuthor>Anonym</RefAuthor> <RefTitle>Verordnung über das Verfahren und die Anforderungen zur Prüfung der Erstattungsfähigkeit digitaler Gesundheitsanwendungen in der gesetzlichen Krankenversicherung (Digitale Gesundheitsanwendungen-Verordnung – DiGAV)</RefTitle> <RefYear>2020</RefYear> <RefJournal>Bundesgesetzblatt</RefJournal> <RefPage>768</RefPage> <RefTotal>Verordnung über das Verfahren und die Anforderungen zur Prüfung der Erstattungsfähigkeit digitaler Gesundheitsanwendungen in der gesetzlichen Krankenversicherung (Digitale Gesundheitsanwendungen-Verordnung – DiGAV). Bundesgesetzblatt. 2020 Apr 8; 2020 Part I(18): 768.</RefTotal> </Reference> <Reference refNo="11"> <RefAuthor>Bundesinstitut für Arzneimittel und Medizinprodukte</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>Das Fast-Track-Verfahren für digitale Gesundheitsanwendungen (DiGA) nach § 139e SGB V. Ein Leitfaden für Hersteller, Leistungserbringer und Anwender</RefBookTitle> <RefPage></RefPage> <RefTotal>Bundesinstitut für Arzneimittel und Medizinprodukte. Das Fast-Track-Verfahren für digitale Gesundheitsanwendungen (DiGA) nach § 139e SGB V. Ein Leitfaden für Hersteller, Leistungserbringer und Anwender. Version 3.1. [last accessed 2023 Feb 02]. Available from: https://www.bfarm.de/SharedDocs/Downloads/DE/Medizinprodukte/diga_leitfaden.pdf?__blob=publicationFile</RefTotal> <RefLink>https://www.bfarm.de/SharedDocs/Downloads/DE/Medizinprodukte/diga_leitfaden.pdf?__blob=publicationFile</RefLink> </Reference> <Reference refNo="12"> <RefAuthor>Brönner M</RefAuthor> <RefAuthor>Meister S</RefAuthor> <RefAuthor>Breil B</RefAuthor> <RefAuthor>Albrecht UV</RefAuthor> <RefTitle>Kapitel 15. Orientierung für Hersteller von Gesundheits-Apps</RefTitle> <RefYear>2016</RefYear> <RefBookTitle>Chancen und Risiken von Gesundheits-Apps (CHARISMHA)</RefBookTitle> <RefPage>320-40</RefPage> <RefTotal>Brönner M, Meister S, Breil B, Albrecht UV. Kapitel 15. Orientierung für Hersteller von Gesundheits-Apps. In: Albrecht UV, editor. Chancen und Risiken von Gesundheits-Apps (CHARISMHA). Medizinische Hochschule Hannover; 2016. p. 320-40.</RefTotal> </Reference> <Reference refNo="13"> <RefAuthor>Bertelsmann Stiftung</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>AppQ. Gütekriterien-Kernset für mehr Qualitätstransparenz bei digitalen Gesundheitsanwendungen</RefBookTitle> <RefPage></RefPage> <RefTotal>Bertelsmann Stiftung. AppQ. Gütekriterien-Kernset für mehr Qualitätstransparenz bei digitalen Gesundheitsanwendungen. [last accessed 2020 Feb 15]. Available from: https://www.bertelsmann-stiftung.de/fileadmin/files/BSt/Publikationen/GrauePublikationen/Studienbericht_AppQ_191028.pdf</RefTotal> <RefLink>https://www.bertelsmann-stiftung.de/fileadmin/files/BSt/Publikationen/GrauePublikationen/Studienbericht_AppQ_191028.pdf</RefLink> </Reference> <Reference refNo="14"> <RefAuthor>Fraunhofer-Institut für Offene Kommunikationssysteme</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>APPKRI Kriterien für Gesundheits-Apps</RefBookTitle> <RefPage></RefPage> <RefTotal>Fraunhofer-Institut für Offene Kommunikationssysteme. APPKRI Kriterien für Gesundheits-Apps. [last accessed 2020 Feb 15]. Available from: https://ehealth-services.fokus.fraunhofer.de/ BMG-APPS/</RefTotal> <RefLink>https://ehealth-services.fokus.fraunhofer.de/ BMG-APPS/</RefLink> </Reference> <Reference refNo="15"> <RefAuthor>NHS England</RefAuthor> <RefTitle>NHS Apps Library. Guidance for health app developers, commissioners and assessors to get trusted digital tools assessed and published on the NHS Apps Library</RefTitle> <RefYear>2020</RefYear> <RefTotal>NHS England. NHS Apps Library. Guidance for health app developers, commissioners and assessors to get trusted digital tools assessed and published on the NHS Apps Library. [last accessed 2020 Feb 15]. Available from: https://digital.nhs.uk/ services/nhs-apps-library</RefTotal> <RefLink>https://digital.nhs.uk/services/nhs-apps-library</RefLink> </Reference> <Reference refNo="16"> <RefAuthor>von Elm E</RefAuthor> <RefAuthor>Schreiber G</RefAuthor> <RefAuthor>Haupt CC</RefAuthor> <RefTitle>Methodische Anleitung für Scoping Reviews (JBI-Methodologie)</RefTitle> <RefYear>2019</RefYear> <RefJournal>Z Evid Fortbild Qual Gesundhwes</RefJournal> <RefPage>1-7</RefPage> <RefTotal>von Elm E, Schreiber G, Haupt CC. Methodische Anleitung für Scoping Reviews (JBI-Methodologie). Z Evid Fortbild Qual Gesundhwes. 2019 Jun;143:1-7. DOI: 10.1016/j.zefq.2019.05.004</RefTotal> <RefLink>https://doi.org/10.1016/j.zefq.2019.05.004</RefLink> </Reference> <Reference refNo="17"> <RefAuthor>Laugesen J</RefAuthor> <RefAuthor>Yuan Y</RefAuthor> <RefTitle>What Factors Contributed to the Success of Apple’s iPhone?</RefTitle> <RefYear></RefYear> <RefBookTitle>Ninth International Conference on Mobile Business and Ninth Global Mobility Roundtable (ICMB-GMR); 2010 June 13-15; Athens</RefBookTitle> <RefPage>91-9</RefPage> <RefTotal>Laugesen J, Yuan Y. What Factors Contributed to the Success of Apple’s iPhone? In: Ninth International Conference on Mobile Business and Ninth Global Mobility Roundtable (ICMB-GMR); 2010 June 13-15; Athens. p. 91-9.</RefTotal> </Reference> <Reference refNo="34"> <RefAuthor>Stoyanov SR</RefAuthor> <RefAuthor>Hides L</RefAuthor> <RefAuthor>Kavanagh DJ</RefAuthor> <RefAuthor>Zelenko O</RefAuthor> <RefAuthor>Tjondronegoro D</RefAuthor> <RefAuthor>Mani M</RefAuthor> <RefTitle>Mobile app rating scale: a new tool for assessing the quality of health mobile apps</RefTitle> <RefYear>2015</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e27</RefPage> <RefTotal>Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015 Mar;3(1):e27. DOI: 10.2196/mhealth.3422</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.3422</RefLink> </Reference> <Reference refNo="35"> <RefAuthor>Stoyanov SR</RefAuthor> <RefAuthor>Hides L</RefAuthor> <RefAuthor>Kavanagh DJ</RefAuthor> <RefAuthor>Wilson H</RefAuthor> <RefTitle>Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS)</RefTitle> <RefYear>2016</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e72</RefPage> <RefTotal>Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth. 2016 Jun;4(2):e72. DOI: 10.2196/mhealth.5849</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.5849</RefLink> </Reference> <Reference refNo="26"> <RefAuthor>Carmody JK</RefAuthor> <RefAuthor>Denson LA</RefAuthor> <RefAuthor>Hommel KA</RefAuthor> <RefTitle>Content and Usability Evaluation of Medication Adherence Mobile Applications for Use in Pediatrics</RefTitle> <RefYear>2019</RefYear> <RefJournal>J Pediatr Psychol</RefJournal> <RefPage>333-42</RefPage> <RefTotal>Carmody JK, Denson LA, Hommel KA. Content and Usability Evaluation of Medication Adherence Mobile Applications for Use in Pediatrics. J Pediatr Psychol. 2019 Apr;44(3):333-42. DOI: 10.1093/jpepsy/jsy086</RefTotal> <RefLink>https://doi.org/10.1093/jpepsy/jsy086</RefLink> </Reference> <Reference refNo="27"> <RefAuthor>Morrissey EC</RefAuthor> <RefAuthor>Corbett TK</RefAuthor> <RefAuthor>Walsh JC</RefAuthor> <RefAuthor>Molloy GJ</RefAuthor> <RefTitle>Behavior Change Techniques in Apps for Medication Adherence: A Content Analysis</RefTitle> <RefYear>2016</RefYear> <RefJournal>Am J Prev Med</RefJournal> <RefPage>e143-6</RefPage> <RefTotal>Morrissey EC, Corbett TK, Walsh JC, Molloy GJ. Behavior Change Techniques in Apps for Medication Adherence: A Content Analysis. Am J Prev Med. 2016 May;50(5):e143-6. DOI: 10.1016/ j.amepre.2015.09.034</RefTotal> <RefLink>https://doi.org/10.1016/j.amepre.2015.09.034</RefLink> </Reference> <Reference refNo="36"> <RefAuthor>Michie S</RefAuthor> <RefAuthor>Richardson M</RefAuthor> <RefAuthor>Johnston M</RefAuthor> <RefAuthor>Abraham C</RefAuthor> <RefAuthor>Francis J</RefAuthor> <RefAuthor>Hardeman W</RefAuthor> <RefAuthor>Eccles MP</RefAuthor> <RefAuthor>Cane J</RefAuthor> <RefAuthor>Wood CE</RefAuthor> <RefTitle>The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions</RefTitle> <RefYear>2013</RefYear> <RefJournal>Ann Behav Med</RefJournal> <RefPage>81-95</RefPage> <RefTotal>Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, Eccles MP, Cane J, Wood CE. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013 Aug;46(1):81-95. DOI: 10.1007/s12160-013-9486-6</RefTotal> <RefLink>https://doi.org/10.1007/s12160-013-9486-6</RefLink> </Reference> <Reference refNo="37"> <RefAuthor>Abraham C</RefAuthor> <RefAuthor>Michie S</RefAuthor> <RefTitle>A taxonomy of behavior change techniques used in interventions</RefTitle> <RefYear>2008</RefYear> <RefJournal>Health Psychol</RefJournal> <RefPage>379-87</RefPage> <RefTotal>Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol. 2008 May;27(3):379-87. DOI: 10.1037/0278-6133.27.3.379</RefTotal> <RefLink>https://doi.org/10.1037/0278-6133.27.3.379</RefLink> </Reference> <Reference refNo="18"> <RefAuthor>Ali EE</RefAuthor> <RefAuthor>Teo AKS</RefAuthor> <RefAuthor>Goh SXL</RefAuthor> <RefAuthor>Chew L</RefAuthor> <RefAuthor>Yap KY</RefAuthor> <RefTitle>MedAd-AppQ: A quality assessment tool for medication adherence apps on iOS and android platforms</RefTitle> <RefYear>2018</RefYear> <RefJournal>Res Social Adm Pharm</RefJournal> <RefPage>1125-33</RefPage> <RefTotal>Ali EE, Teo AKS, Goh SXL, Chew L, Yap KY. MedAd-AppQ: A quality assessment tool for medication adherence apps on iOS and android platforms. Res Social Adm Pharm. 2018 Dec;14(12): 1125-33. DOI: 10.1016/j.sapharm.2018.01.006</RefTotal> <RefLink>https://doi.org/10.1016/j.sapharm.2018.01.006</RefLink> </Reference> <Reference refNo="39"> <RefAuthor>Huckvale K</RefAuthor> <RefAuthor>Adomaviciute S</RefAuthor> <RefAuthor>Prieto JT</RefAuthor> <RefAuthor>Leow MK</RefAuthor> <RefAuthor>Car J</RefAuthor> <RefTitle>Smartphone apps for calculating insulin dose: a systematic assessment</RefTitle> <RefYear>2015</RefYear> <RefJournal>BMC Med</RefJournal> <RefPage>106</RefPage> <RefTotal>Huckvale K, Adomaviciute S, Prieto JT, Leow MK, Car J. Smartphone apps for calculating insulin dose: a systematic assessment. BMC Med. 2015 May;13:106. DOI: 10.1186/ s12916-015-0314-7</RefTotal> <RefLink>https://doi.org/10.1186/s12916-015-0314-7</RefLink> </Reference> <Reference refNo="40"> <RefAuthor>Lewis TL</RefAuthor> <RefTitle>A systematic self-certification model for mobile medical apps</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Med Internet Res</RefJournal> <RefPage>e89</RefPage> <RefTotal>Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res. 2013 Apr;15(4):e89. DOI: 10.2196/ jmir.2446</RefTotal> <RefLink>https://doi.org/10.2196/jmir.2446</RefLink> </Reference> <Reference refNo="20"> <RefAuthor>Dayer L</RefAuthor> <RefAuthor>Heldenbrand S</RefAuthor> <RefAuthor>Anderson P</RefAuthor> <RefAuthor>Gubbins PO</RefAuthor> <RefAuthor>Martin BC</RefAuthor> <RefTitle>Smartphone medication adherence apps: potential benefits to patients and providers</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Am Pharm Assoc (2003)</RefJournal> <RefPage>172-81</RefPage> <RefTotal>Dayer L, Heldenbrand S, Anderson P, Gubbins PO, Martin BC. Smartphone medication adherence apps: potential benefits to patients and providers. J Am Pharm Assoc (2003). 2013 Mar-Apr;53(2):172-81. DOI: 10.1331/JAPhA.2013.12202</RefTotal> <RefLink>https://doi.org/10.1331/JAPhA.2013.12202</RefLink> </Reference> <Reference refNo="28"> <RefAuthor>Dayer LE</RefAuthor> <RefAuthor>Shilling R</RefAuthor> <RefAuthor>Van Valkenburg M</RefAuthor> <RefAuthor>Martin BC</RefAuthor> <RefAuthor>Gubbins PO</RefAuthor> <RefAuthor>Hadden K</RefAuthor> <RefAuthor>Heldenbrand S</RefAuthor> <RefTitle>Assessing the Medication Adherence App Marketplace From the Health Professional and Consumer Vantage Points</RefTitle> <RefYear>2017</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e45</RefPage> <RefTotal>Dayer LE, Shilling R, Van Valkenburg M, Martin BC, Gubbins PO, Hadden K, Heldenbrand S. Assessing the Medication Adherence App Marketplace From the Health Professional and Consumer Vantage Points. JMIR Mhealth Uhealth. 2017 Apr;5(4):e45. DOI: 10.2196/mhealth.6582</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.6582</RefLink> </Reference> <Reference refNo="21"> <RefAuthor>Heldenbrand S</RefAuthor> <RefAuthor>Martin BC</RefAuthor> <RefAuthor>Gubbins PO</RefAuthor> <RefAuthor>Hadden K</RefAuthor> <RefAuthor>Renna C</RefAuthor> <RefAuthor>Shilling R</RefAuthor> <RefAuthor>Dayer L</RefAuthor> <RefTitle>Assessment of medication adherence app features, functionality, and health literacy level and the creation of a searchable Web-based adherence app resource for health care professionals and patients</RefTitle> <RefYear>2016</RefYear> <RefJournal>J Am Pharm Assoc (2003)</RefJournal> <RefPage>293-302</RefPage> <RefTotal>Heldenbrand S, Martin BC, Gubbins PO, Hadden K, Renna C, Shilling R, Dayer L. Assessment of medication adherence app features, functionality, and health literacy level and the creation of a searchable Web-based adherence app resource for health care professionals and patients. J Am Pharm Assoc (2003). 2016 May-Jun;56(3):293-302. DOI: 10.1016/j.japh.2015.12.014</RefTotal> <RefLink>https://doi.org/10.1016/j.japh.2015.12.014</RefLink> </Reference> <Reference refNo="29"> <RefAuthor>Grindrod KA</RefAuthor> <RefAuthor>Li M</RefAuthor> <RefAuthor>Gates A</RefAuthor> <RefTitle>Evaluating user perceptions of mobile medication management applications with older adults: a usability study</RefTitle> <RefYear>2014</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e11</RefPage> <RefTotal>Grindrod KA, Li M, Gates A. Evaluating user perceptions of mobile medication management applications with older adults: a usability study. JMIR Mhealth Uhealth. 2014 Mar;2(1):e11. DOI: 10.2196/mhealth.3048</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.3048</RefLink> </Reference> <Reference refNo="41"> <RefAuthor>Brooke J</RefAuthor> <RefTitle>SUS: a “quick and dirty” usability scale</RefTitle> <RefYear>1996</RefYear> <RefBookTitle>Usability evaluation in industry</RefBookTitle> <RefPage></RefPage> <RefTotal>Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, editors. Usability evaluation in industry. London: Taylor and Francis; 1996.</RefTotal> </Reference> <Reference refNo="33"> <RefAuthor>Grindrod K</RefAuthor> <RefAuthor>Boersema J</RefAuthor> <RefAuthor>Waked K</RefAuthor> <RefAuthor>Smith V</RefAuthor> <RefAuthor>Yang J</RefAuthor> <RefAuthor>Gebotys C</RefAuthor> <RefTitle>Locking it down: The privacy and security of mobile medication apps</RefTitle> <RefYear>2017</RefYear> <RefJournal>Can Pharm J (Ott)</RefJournal> <RefPage>60-6</RefPage> <RefTotal>Grindrod K, Boersema J, Waked K, Smith V, Yang J, Gebotys C. Locking it down: The privacy and security of mobile medication apps. Can Pharm J (Ott). 2017 Jan-Feb;150(1):60-6. DOI: 10.1177/1715163516680226</RefTotal> <RefLink>https://doi.org/10.1177/1715163516680226</RefLink> </Reference> <Reference refNo="30"> <RefAuthor>Jupp JCY</RefAuthor> <RefAuthor>Sultani H</RefAuthor> <RefAuthor>Cooper CA</RefAuthor> <RefAuthor>Peterson KA</RefAuthor> <RefAuthor>Truong TH</RefAuthor> <RefTitle>Evaluation of mobile phone applications to support medication adherence and symptom management in oncology patients</RefTitle> <RefYear>2018</RefYear> <RefJournal>Pediatr Blood Cancer</RefJournal> <RefPage>e27278</RefPage> <RefTotal>Jupp JCY, Sultani H, Cooper CA, Peterson KA, Truong TH. Evaluation of mobile phone applications to support medication adherence and symptom management in oncology patients. Pediatr Blood Cancer. 2018 Nov;65(11):e27278. DOI: 10.1002/pbc.27278</RefTotal> <RefLink>https://doi.org/10.1002/pbc.27278</RefLink> </Reference> <Reference refNo="22"> <RefAuthor>Santo K</RefAuthor> <RefAuthor>Richtering SS</RefAuthor> <RefAuthor>Chalmers J</RefAuthor> <RefAuthor>Thiagalingam A</RefAuthor> <RefAuthor>Chow CK</RefAuthor> <RefAuthor>Redfern J</RefAuthor> <RefTitle>Mobile Phone Apps to Improve Medication Adherence: A Systematic Stepwise Process to Identify High-Quality Apps</RefTitle> <RefYear>2016</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e132</RefPage> <RefTotal>Santo K, Richtering SS, Chalmers J, Thiagalingam A, Chow CK, Redfern J. Mobile Phone Apps to Improve Medication Adherence: A Systematic Stepwise Process to Identify High-Quality Apps. JMIR Mhealth Uhealth. 2016 Dec;4(4):e132. DOI: 10.2196/ mhealth.6742</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.6742</RefLink> </Reference> <Reference refNo="31"> <RefAuthor>Kim BY</RefAuthor> <RefAuthor>Sharafoddini A</RefAuthor> <RefAuthor>Tran N</RefAuthor> <RefAuthor>Wen EY</RefAuthor> <RefAuthor>Lee J</RefAuthor> <RefTitle>Consumer Mobile Apps for Potential Drug-Drug Interaction Check: Systematic Review and Content Analysis Using the Mobile App Rating Scale (MARS)</RefTitle> <RefYear>2018</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e74</RefPage> <RefTotal>Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer Mobile Apps for Potential Drug-Drug Interaction Check: Systematic Review and Content Analysis Using the Mobile App Rating Scale (MARS). JMIR Mhealth Uhealth. 2018 Mar;6(3):e74. DOI: 10.2196/mhealth.8613</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.8613</RefLink> </Reference> <Reference refNo="32"> <RefAuthor>Liu Y</RefAuthor> <RefAuthor>Chang P</RefAuthor> <RefAuthor>Li N</RefAuthor> <RefAuthor>Lamb KV</RefAuthor> <RefAuthor>Wang J</RefAuthor> <RefAuthor>Huang X</RefAuthor> <RefAuthor>Wang Y</RefAuthor> <RefAuthor>Wu Y</RefAuthor> <RefTitle>Quality of Chinese CVD Medication Apps</RefTitle> <RefYear>2016</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>1036-7</RefPage> <RefTotal>Liu Y, Chang P, Li N, Lamb KV, Wang J, Huang X, Wang Y, Wu Y. Quality of Chinese CVD Medication Apps. Stud Health Technol Inform. 2016;225:1036-7.</RefTotal> </Reference> <Reference refNo="42"> <RefAuthor>Jeon E</RefAuthor> <RefAuthor>Park HA</RefAuthor> <RefAuthor>Min YH</RefAuthor> <RefAuthor>Kim HY</RefAuthor> <RefTitle>Analysis of the information quality of korean obesity-management smartphone applications</RefTitle> <RefYear>2014</RefYear> <RefJournal>Healthc Inform Res</RefJournal> <RefPage>23-9</RefPage> <RefTotal>Jeon E, Park HA, Min YH, Kim HY. Analysis of the information quality of korean obesity-management smartphone applications. Healthc Inform Res. 2014 Jan;20(1):23-9. DOI: 10.4258/ hir.2014.20.1.23</RefTotal> <RefLink>https://doi.org/10.4258/hir.2014.20.1.23</RefLink> </Reference> <Reference refNo="19"> <RefAuthor>Loy JS</RefAuthor> <RefAuthor>Ali EE</RefAuthor> <RefAuthor>Yap KY</RefAuthor> <RefTitle>Quality Assessment of Medical Apps that Target Medication-Related Problems</RefTitle> <RefYear>2016</RefYear> <RefJournal>J Manag Care Spec Pharm</RefJournal> <RefPage>1124-40</RefPage> <RefTotal>Loy JS, Ali EE, Yap KY. Quality Assessment of Medical Apps that Target Medication-Related Problems. J Manag Care Spec Pharm. 2016 Oct;22(10):1124-40. DOI: 10.18553/jmcp.2016.22.10.1124</RefTotal> <RefLink>https://doi.org/10.18553/jmcp.2016.22.10.1124</RefLink> </Reference> <Reference refNo="43"> <RefAuthor>Anonym</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>iMedicalApps. iMedicalApps Medpage Today</RefBookTitle> <RefPage></RefPage> <RefTotal>iMedicalApps. iMedicalApps Medpage Today. [last accessed 2020 Mar 03]. Available from: https://www.imedicalapps.com/</RefTotal> <RefLink>https://www.imedicalapps.com/</RefLink> </Reference> <Reference refNo="23"> <RefAuthor>Nguyen E</RefAuthor> <RefAuthor>Bugno L</RefAuthor> <RefAuthor>Kandah C</RefAuthor> <RefAuthor>Plevinsky J</RefAuthor> <RefAuthor>Poulopoulos N</RefAuthor> <RefAuthor>Wojtowicz A</RefAuthor> <RefAuthor>Schneider KL</RefAuthor> <RefAuthor>Greenley RN</RefAuthor> <RefTitle>Is There a Good App for That? Evaluating m-Health Apps for Strategies That Promote Pediatric Medication Adherence</RefTitle> <RefYear>2016</RefYear> <RefJournal>Telemed J E Health</RefJournal> <RefPage>929-37</RefPage> <RefTotal>Nguyen E, Bugno L, Kandah C, Plevinsky J, Poulopoulos N, Wojtowicz A, Schneider KL, Greenley RN. Is There a Good App for That? Evaluating m-Health Apps for Strategies That Promote Pediatric Medication Adherence. Telemed J E Health. 2016 11;22(11):929-37. DOI: 10.1089/tmj.2015.0211</RefTotal> <RefLink>https://doi.org/10.1089/tmj.2015.0211</RefLink> </Reference> <Reference refNo="44"> <RefAuthor>Kahana S</RefAuthor> <RefAuthor>Drotar D</RefAuthor> <RefAuthor>Frazier T</RefAuthor> <RefTitle>Meta-analysis of psychological interventions to promote adherence to treatment in pediatric chronic health conditions</RefTitle> <RefYear>2008</RefYear> <RefJournal>J Pediatr Psychol</RefJournal> <RefPage>590-611</RefPage> <RefTotal>Kahana S, Drotar D, Frazier T. Meta-analysis of psychological interventions to promote adherence to treatment in pediatric chronic health conditions. J Pediatr Psychol. 2008 Jul;33(6):590-611. DOI: 10.1093/jpepsy/jsm128</RefTotal> <RefLink>https://doi.org/10.1093/jpepsy/jsm128</RefLink> </Reference> <Reference refNo="45"> <RefAuthor>Pai AL</RefAuthor> <RefAuthor>McGrady M</RefAuthor> <RefTitle>Systematic review and meta-analysis of psychological interventions to promote treatment adherence in children, adolescents, and young adults with chronic illness</RefTitle> <RefYear>2014</RefYear> <RefJournal>J Pediatr Psychol</RefJournal> <RefPage>918-31</RefPage> <RefTotal>Pai AL, McGrady M. Systematic review and meta-analysis of psychological interventions to promote treatment adherence in children, adolescents, and young adults with chronic illness. J Pediatr Psychol. 2014 Sep;39(8):918-31. DOI: 10.1093/ jpepsy/jsu038</RefTotal> <RefLink>https://doi.org/10.1093/jpepsy/jsu038</RefLink> </Reference> <Reference refNo="24"> <RefAuthor>Park JYE</RefAuthor> <RefAuthor>Li J</RefAuthor> <RefAuthor>Howren A</RefAuthor> <RefAuthor>Tsao NW</RefAuthor> <RefAuthor>De Vera M</RefAuthor> <RefTitle>Mobile Phone Apps Targeting Medication Adherence: Quality Assessment and Content Analysis of User Reviews</RefTitle> <RefYear>2019</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e11919</RefPage> <RefTotal>Park JYE, Li J, Howren A, Tsao NW, De Vera M. Mobile Phone Apps Targeting Medication Adherence: Quality Assessment and Content Analysis of User Reviews. JMIR Mhealth Uhealth. 2019 Jan;7(1):e11919. DOI: 10.2196/11919</RefTotal> <RefLink>https://doi.org/10.2196/11919</RefLink> </Reference> <Reference refNo="46"> <RefAuthor>Stawarz K</RefAuthor> <RefAuthor>Cox A</RefAuthor> <RefAuthor>Blandford A</RefAuthor> <RefTitle>Don’t forget your pill! Designing effective medication reminder apps that support users’ daily routines</RefTitle> <RefYear>2014</RefYear> <RefBookTitle>32nd annual Association for Computing Machinery (ACM) conference on Human factors in computing systems; 2014 Apr 26 – May 01; Toronto, Ontario, Canada</RefBookTitle> <RefPage>2269-78</RefPage> <RefTotal>Stawarz K, Cox A, Blandford A. Don’t forget your pill! Designing effective medication reminder apps that support users’ daily routines. In: 32nd annual Association for Computing Machinery (ACM) conference on Human factors in computing systems; 2014 Apr 26 – May 01; Toronto, Ontario, Canada; 2014. p. 2269-78.</RefTotal> </Reference> <Reference refNo="47"> <RefAuthor>Bailey SC</RefAuthor> <RefAuthor>Belter LT</RefAuthor> <RefAuthor>Pandit AU</RefAuthor> <RefAuthor>Carpenter DM</RefAuthor> <RefAuthor>Carlos E</RefAuthor> <RefAuthor>Wolf MS</RefAuthor> <RefTitle>The availability, functionality, and quality of mobile applications supporting medication self-management</RefTitle> <RefYear>2014</RefYear> <RefJournal>J Am Med Inform Assoc</RefJournal> <RefPage>542-6</RefPage> <RefTotal>Bailey SC, Belter LT, Pandit AU, Carpenter DM, Carlos E, Wolf MS. The availability, functionality, and quality of mobile applications supporting medication self-management. J Am Med Inform Assoc. 2014 May-Jun;21(3):542-6. DOI: 10.1136/amiajnl-2013-002232</RefTotal> <RefLink>https://doi.org/10.1136/amiajnl-2013-002232</RefLink> </Reference> <Reference refNo="25"> <RefAuthor>Tabi K</RefAuthor> <RefAuthor>Randhawa AS</RefAuthor> <RefAuthor>Choi F</RefAuthor> <RefAuthor>Mithani Z</RefAuthor> <RefAuthor>Albers F</RefAuthor> <RefAuthor>Schnieder M</RefAuthor> <RefAuthor>Nikoo M</RefAuthor> <RefAuthor>Vigo D</RefAuthor> <RefAuthor>Jang K</RefAuthor> <RefAuthor>Demlova R</RefAuthor> <RefAuthor>Krausz M</RefAuthor> <RefTitle>Mobile Apps for Medication Management: Review and Analysis</RefTitle> <RefYear>2019</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e13608</RefPage> <RefTotal>Tabi K, Randhawa AS, Choi F, Mithani Z, Albers F, Schnieder M, Nikoo M, Vigo D, Jang K, Demlova R, Krausz M. Mobile Apps for Medication Management: Review and Analysis. JMIR Mhealth Uhealth. 2019 Sep;7(9):e13608. DOI: 10.2196/13608</RefTotal> <RefLink>https://doi.org/10.2196/13608</RefLink> </Reference> <Reference refNo="48"> <RefAuthor>Shen N</RefAuthor> <RefAuthor>Levitan MJ</RefAuthor> <RefAuthor>Johnson A</RefAuthor> <RefAuthor>Bender JL</RefAuthor> <RefAuthor>Hamilton-Page M</RefAuthor> <RefAuthor>Jadad AA</RefAuthor> <RefAuthor>Wiljer D</RefAuthor> <RefTitle>Finding a depression app: a review and content analysis of the depression app marketplace</RefTitle> <RefYear>2015</RefYear> <RefJournal>JMIR Mhealth Uhealth</RefJournal> <RefPage>e16</RefPage> <RefTotal>Shen N, Levitan MJ, Johnson A, Bender JL, Hamilton-Page M, Jadad AA, Wiljer D. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth. 2015 Feb;3(1):e16. DOI: 10.2196/ mhealth.3713</RefTotal> <RefLink>https://doi.org/10.2196/mhealth.3713</RefLink> </Reference> <Reference refNo="49"> <RefAuthor>Bender JL</RefAuthor> <RefAuthor>Yue RY</RefAuthor> <RefAuthor>To MJ</RefAuthor> <RefAuthor>Deacken L</RefAuthor> <RefAuthor>Jadad AR</RefAuthor> <RefTitle>A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Med Internet Res</RefJournal> <RefPage>e287</RefPage> <RefTotal>Bender JL, Yue RY, To MJ, Deacken L, Jadad AR. A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer. J Med Internet Res. 2013 Dec;15 (12):e287. DOI: 10.2196/jmir.2661</RefTotal> <RefLink>https://doi.org/10.2196/jmir.2661</RefLink> </Reference> <Reference refNo="38"> <RefAuthor>Arah OA</RefAuthor> <RefAuthor>Westert GP</RefAuthor> <RefAuthor>Hurst J</RefAuthor> <RefAuthor>Klazinga NS</RefAuthor> <RefTitle>A conceptual framework for the OECD Health Care Quality Indicators Project</RefTitle> <RefYear>2006</RefYear> <RefJournal>Int J Qual Health Care</RefJournal> <RefPage>5-13</RefPage> <RefTotal>Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual framework for the OECD Health Care Quality Indicators Project. Int J Qual Health Care. 2006 Sep;18 Suppl 1:5-13. DOI: 10.1093/intqhc/mzl024</RefTotal> <RefLink>https://doi.org/10.1093/intqhc/mzl024</RefLink> </Reference> </References> <Media> <Tables> <Table format="png"> <MediaNo>1</MediaNo> <MediaID>1</MediaID> <Caption><Pgraph><Mark1>Table 1: Overview of the studies included in this scoping review</Mark1></Pgraph></Caption> </Table> <NoOfTables>1</NoOfTables> </Tables> <Figures> <Figure format="png" height="859" width="941"> <MediaNo>1</MediaNo> <MediaID>1</MediaID> <Caption><Pgraph><Mark1>Figure 1: Flowchart of the scoping review</Mark1></Pgraph></Caption> </Figure> <Figure format="png" height="883" width="1262"> <MediaNo>2</MediaNo> <MediaID>2</MediaID> <Caption><Pgraph><Mark1>Figure 2: Summary of the evaluation criteria and categorization. Criteria mentioned in only one study are not included in the figure. These include “technical support” (user friendliness), “data transfer to the ePA” (interoperability/collaboration), “recording progress”, “problem solving regarding adherence barriers” (motivation to change behavior), “risk assessment”, “emergency management” (medication/patient safety), and “reminder of physician appointments” (reminder).</Mark1></Pgraph></Caption> </Figure> <NoOfPictures>2</NoOfPictures> </Figures> <InlineFigures> <NoOfPictures>0</NoOfPictures> </InlineFigures> <Attachments> <Attachment> <MediaNo>1</MediaNo> <MediaID filename="000317.a1.pdf" mimeType="application/pdf" origFilename="Attachment 1_gms000317.pdf" size="107953" url="">1</MediaID> <AttachmentTitle>Appendix 1–3</AttachmentTitle> </Attachment> <Attachment> <MediaNo>2</MediaNo> <MediaID filename="000317.a2.pdf" mimeType="application/pdf" origFilename="Attachment 2_gms000317.pdf" size="118416" url="">2</MediaID> <AttachmentTitle>Appendix 4</AttachmentTitle> </Attachment> <NoOfAttachments>2</NoOfAttachments> </Attachments> </Media> </OrigData> </GmsArticle>