mibe000199 10.3205/mibe000199 urn:nbn:de:0183-mibe0001998 Research Article Indicators of data quality: review and requirements from the perspective of networked medical research Indikatoren zur Datenqualität: Stand und Anforderungen aus Sicht der vernetzten medizinischen Forschung Stausberg Stausberg Jürgen J Prof. Dr. med.
Institute for Medical Informatics, Biometry and Epidemiology, Faculty of Medicine, University Duisburg-Essen, Hufelandstrasse 55, 45122 Essen, GermanyInstitute for Medical Informatics, Biometry and Epidemiology (IMIBE), Faculty of Medicine, University Duisburg-Essen, Germany
stausberg@ekmed.de author Bauer Bauer Ulrike U
Competence Network for Congenital Heart Defects, German Centre for Cardiovascular Research, Berlin, Germany
author Nasseh Nasseh Daniel D
Institut für medizinische Informationsverarbeitung, Biometrie und Epidemiologie (IBE), Ludwig-Maximilians-Universität München, Germany
author Pritzkuleit Pritzkuleit Ron R
Institute for Cancer Epidemiology e.V., University Lübeck, Germany
author Schmidt Schmidt Carsten O. CO
Study of Health in Pomerania, Institute for Community Medicine, Greifswald, Germany
author Schrader Schrader Thomas T
Resource Center OpEN.SC, Brandenburg University of Applied Sciences, Brandenburg, Germany
author Nonnemacher Nonnemacher Michael M
Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), Faculty of Medicine, University Duisburg-Essen, Germany
author German Medical Science GMS Publishing House
Düsseldorf
 610 medical research data quality healthcare guidelines analytics informatics medizinische Forschung Datenqualität Gesundheitswesen Leitlinie Analyse Informatik 20190709 engl This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung). 1860-9171 15 1 GMS Medizinische Informatik, Biometrie und Epidemiologie GMS Med Inform Biom Epidemiol 05 TMF DFG Datenqualität ist für die quantitative medizinische Forschung von höchster Bedeutung. Ein einheitliches Set von Indikatoren zur Datenqualität wird benötigt, um die zukünftigen Herausforderungen an das Datenmanagement in der biomedizinischen Informatik zu bewältigen. Dazu wurde eine Leitlinie zum adaptiven Datenmanagement im Jahre 2006 erarbeitet, die Indikatoren zur Datenqualität über drei Ebenen organisiert: die Ebenen Integrität, Organisation und Richtigkeit. Inhaltlich wurde die Leitlinie im Jahre 2014 Bottom-up durch die Einbindung von Standards eines Krebsregisters, einer Kohorte und eines Data Repository aus Deutschland erweitert. Parallel wurden über ein systematisches Literaturreview publizierte Indikatoren der Datenqualität mit Medline als Literaturdatenbank recherchiert. Die Leitlinie weist in ihrer zweiten Version 51 Indikatoren aus (Integrität: 30, Organisation: 15, Richtigkeit: 6). Das Literaturreview identifizierte 34 Indikatoren in 31 Publikationen. Im Vergleich beider Quellen war das Fehlen von Indikatoren zu organisatorischen Aspekten in der Literatur auffällig. Der Leitlinie fehlten hingegen Indikatoren mit Bedeutung für die Krankenversorgung wie Rechtzeitigkeit. Der vorgenommene Vergleich stellt einen weiteren Schritt zur Festlegung einem einheitlichen Sets von Indikatoren zur Datenqualität in der medizinischen Forschung dar. Neben einheitlichen Bezeichnungen sollte ein solches Set umsetzbare Definitionen beinhalten, die eine zuverlässige Anwendung auf unterschiedlichen Datenbeständen durch unterschiedliche Forschergruppen sicherstellt. Zusätzlich würde eine systematische Organisation der Indikatoren eine angemessene Auswahl von Indikatoren für unterschiedliche Anwendungsszenarien unterstützen. Data quality is of highest importance for quantitative medical research. A common set of indicators for data quality is needed to cope with the future challenges in data management for biomedical informatics. A guideline for adaptive data management was developed in 2006, which offers indicators for data quality organized in three categories: integrity, organization, and trueness. The guideline was revised in 2014 bottom-up by extending its content with standards from a cancer registry, a cohort, and a data repository in Germany. In parallel, a systematic literature review identified indicators of data quality published in the literature since 2005 using Medline as literature database. The guideline differentiates in its second version 51 indicators (integrity: 30, organization: 1</PlainText></TextGroup>5, trueness: 6). The literature review identified 34 indicators in <TextGroup><PlainText>31 a</PlainText></TextGroup>rticles. A lack of indicators in the literature addressing the organizational aspects of data sets became visible comparing both sets. Furthermore, indicators useful for data sets used in health care practice, such as timeliness, were missing in the guideline&#8217;s set. The comparison is a first step towards a common set of indicators. Beyond a consented denomination of the indicators, this set should offer an operational definition that supports a reliable application from different parties to different data sets. Furthermore, a systematic organization of the indi<TextGroup><PlainText>cators</PlainText></TextGroup> would foster an appropriate selection of the individual indicators according to specific use cases.</Pgraph></Abstract> <TextBlock linked="yes" name="Introduction"> <MainHeadline>Introduction</MainHeadline><Pgraph>Data are the treasure of quantitative research. Strenuous efforts are undertaken to obtain high-quality data <TextLink reference="1"></TextLink>. Metadata are defined, data acquisition is standardized, data collection is supported by plausibility checks, data quality is reported to study sites, recorded data are compared to originals ones, to mention only some of the available methods to achieve high data quality. However, those methods are only beneficial if their success is controlled. Moreover, those methods could be tailored to the level of the assessed data quality <TextLink reference="2"></TextLink>. </Pgraph><Pgraph>Data are more and more used beyond their original context, for example data from the electronic patient record in clinical trials <TextLink reference="3"></TextLink>. Then, an assessment of the data quality is needed to decide whether the data are appropriate to answer a specific research question or not <TextLink reference="4"></TextLink>. The use of indicators or key performance measures is an established methodology in health care to assess quality <TextLink reference="5"></TextLink>. Results that are closer to a predefined goal or closer to an optimum indicate better quality. Meanwhile, the use of quality indicators becomes accepted also for the assessment of data quality. For example, cancer registries have a long tradition in calculating measures such as case completeness, data completeness and validity <TextLink reference="6"></TextLink>.</Pgraph><Pgraph>There is a strong emphasis on synthesizing a conceptual framework covering terminological and ontological aspects in data quality research. Wang and Strong distinguished four dimensions of data quality through a systematic approach; intrinsic data quality, contextual data quality, representational data quality, and accessibility data quality <TextLink reference="7"></TextLink>. Fifteen indicators were assigned to one dimension each and briefly described by a single sentence. However, this work was not really elaborated in view of health care. Botsis et al. reduced data quality to the aspects of incompleteness (i.e. missing information), inconsistency (i.e. information mismatch), and inaccuracy <TextLink reference="8"></TextLink>. Weiskopf and Weng came up with completeness, correctness, concordance, plausibility, and currency as dimensions <TextLink reference="9"></TextLink>. They defined currency as &#8220;a relevant representation of the patient state at a given point in time&#8221;. Recency and timeliness were listed as related terms. Furthermore, Weiskopf and Weng extended the perspective of data quality dimensions with seven data quality assessment methods like data source agreement. Kahn et al. shortened the top-level dimensions of Weiskopf and Weng to the data quality categories conformance, completeness, and plausibility related to the data quality assessment contexts verification and validation <TextLink reference="10"></TextLink>.</Pgraph><Pgraph>More than ten years ago, a group within the TMF &#8211; Technology, Methods, and Infrastructure for Networked Medical Research, an umbrella organization for networked medical research in Germany, developed a guideline for an adaptive management of data quality <TextLink reference="2"></TextLink>, <TextLink reference="11"></TextLink>. This work started with the aim to support the practice of data management, in opposite to the conceptual approaches introduced before. The methodology applied was influenced by quality research, in particular health care quality research. According to Donabedian, </Pgraph><Pgraph><OrderedList><ListItem level="1" levelPosition="1" numString="1.">quality can be described on the levels of structures, processes, and outcomes <TextLink reference="12"></TextLink>, </ListItem><ListItem level="1" levelPosition="2" numString="2.">quality is measured using indicators <TextLink reference="5"></TextLink>, and </ListItem><ListItem level="1" levelPosition="3" numString="3.">continuous quality improvement is driven by the quality circle of Deming <TextLink reference="13"></TextLink>. </ListItem></OrderedList></Pgraph><Pgraph>Central to the guideline is, first, the measurement of data quality using a set of indicators, and second adapting source data verification and feedback to the level of data quality that becomes evident by the indicator results. The first version of the guideline, published in 2006, included 24 indicators organized in three categories: plausibility (10 indicators), organization (7), and trueness (7) <TextLink reference="11"></TextLink>. Plausibility referred to Donabedian&#8217;s level of structures, organization to the level of processes, and trueness to the level of outcomes. The indicators were identified based on a systematic literature review. Due to the focus of the TMF members, the guideline particularly addressed the needs of cohorts and registries. </Pgraph><Pgraph>In the revision of the guideline a different approach was applied <TextLink reference="14"></TextLink>. On the one hand, the list of indicators was evaluated and extended bottom-up making use of real-world examples of cohorts, data repositories, and registries. On the other hand, the systematic literature review concerning data quality was updated in parallel. Objective of the current study was to compare both results to find gaps that could be closed in future work and to identify consensus that could help to establish a consistent and unambiguous matrix of indicators of data quality.</Pgraph></TextBlock> <TextBlock linked="yes" name="Material and methods"> <MainHeadline>Material and methods</MainHeadline><SubHeadline>Guideline revision</SubHeadline><Pgraph>A bottom-up approach was applied in order to evaluate and update the list of quality indicators <TextLink reference="14"></TextLink>. Projects were identified that are suited as proxies for cohorts, registries, and data repositories representing the main types of quantitative research of the TMF members. The measures for data quality used by those projects were collected and mapped onto the list of indicators defined in the first version of the guideline. Measures missing in the guideline were added based on a consensual decision by the study participants.</Pgraph><Pgraph>The proxies were as follows.</Pgraph><Pgraph><UnorderedList><ListItem level="1">One statutory epidemiological cancer registry participated as proxy for registries, being an active member of the Association of Population-based Cancer Registries in Germany (GEKID).</ListItem><ListItem level="1">The Study of Health in Pomerania (SHIP) participated as a proxy for cohort studies. SHIP is a large population-based epidemiological study in the region of Western Pomerania, Germany.</ListItem><ListItem level="1">The Open European Nephrology Science Center (OpEN.SC) represented data repositories. Data repositories collect data from a wide range of studies without a predefined research question. The data are then provided to third parties.</ListItem></UnorderedList></Pgraph><SubHeadline>Systematic literature review</SubHeadline><Pgraph>The literature review followed a standardized approach according to the recommendations on Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) <TextLink reference="15"></TextLink>. Medline was used as literature database. Citations from 2005 to March 2013 in English and German were included. The queries covered the following terms in several combinations: clinical trial, cohort, data accuracy, data collection, data quality, feedback, fraud, medical registry, quality assessment, quality control, registries, and source data verification. The selection of the relevant literature was conducted in two steps and controlled by an overlapping evaluation between three raters. Decisions in case of questionable citations were made in a consensus. The indicators from the systematic literature review were mapped to the guideline&#8217;s list of indicators. Denominations and definitions were used both for the mapping.</Pgraph></TextBlock> <TextBlock linked="yes" name="Results"> <MainHeadline>Results</MainHeadline><SubHeadline>Quality indicators proposed by the guideline</SubHeadline><Pgraph>The second version of the guideline was expanded to 51 quality indicators organized in three categories: integrity (former denomination plausibility, 30 indicators), organization (15) and trueness (6) <TextLink reference="16"></TextLink>. For the first time, an indicator addressing the quality of metadata was included. This indicator belongs to the category integrity. In the second version, the structured description of each indicator was substituted by information about the appropriate context. Three possibilities were differentiated for that context: </Pgraph><Pgraph><OrderedList><ListItem level="1" levelPosition="1" numString="1.">an indicator can be calculated for an individual record, </ListItem><ListItem level="1" levelPosition="2" numString="2.">an indicator can be calculated for an individual observational unit, </ListItem><ListItem level="1" levelPosition="3" numString="3.">an indicator can be calculated for a complete data set. </ListItem></OrderedList></Pgraph><Pgraph>Context 3 is the traditional one in the application of indicators. Table 1 <ImgLink imgNo="1" imgType="table"/> shows the list of all 51 indicators. Each quality indicator is defined in a structured format dating back to recommendations of the Joint Commission on Accreditation of Healthcare Organisations (JCAHO) <TextLink reference="5"></TextLink> using the following attributes: name, description, definition of terms, identifier, type of indicator (structure, process or outcome), literature references, context (see above), alternative definitions, comments, numerator, denominator, subcategories, method of calculation, interpretation of results, predictors and confounders (cf. Attachment 1 <AttachmentLink attachmentNo="1"/> for an example).</Pgraph><SubHeadline>Quality indicators identified in the literature review</SubHeadline><Pgraph>The systematic literature review yielded 39 articles concerned with either indicators of data quality, feedback about data quality, or source data verification <TextLink reference="7"></TextLink>, <TextLink reference="9"></TextLink>, <TextLink reference="17"></TextLink>, <TextLink reference="18"></TextLink>, <TextLink reference="19"></TextLink>, <TextLink reference="20"></TextLink>, <TextLink reference="21"></TextLink>, <TextLink reference="22"></TextLink>, <TextLink reference="23"></TextLink>, <TextLink reference="24"></TextLink>, <TextLink reference="25"></TextLink>, <TextLink reference="26"></TextLink>, <TextLink reference="27"></TextLink>, <TextLink reference="28"></TextLink>, <TextLink reference="29"></TextLink>, <TextLink reference="30"></TextLink>, <TextLink reference="31"></TextLink>, <TextLink reference="32"></TextLink>, <TextLink reference="33"></TextLink>, <TextLink reference="34"></TextLink>, <TextLink reference="35"></TextLink>, <TextLink reference="36"></TextLink>, <TextLink reference="37"></TextLink>, <TextLink reference="38"></TextLink>, <TextLink reference="39"></TextLink>, <TextLink reference="40"></TextLink>, <TextLink reference="41"></TextLink>, <TextLink reference="42"></TextLink>, <TextLink reference="43"></TextLink>, <TextLink reference="44"></TextLink>, <TextLink reference="45"></TextLink>, <TextLink reference="46"></TextLink>, <TextLink reference="47"></TextLink>, <TextLink reference="48"></TextLink>, <TextLink reference="49"></TextLink>, <TextLink reference="50"></TextLink>, <TextLink reference="51"></TextLink>, <TextLink reference="52"></TextLink>, <TextLink reference="53"></TextLink>. Thirty-one of the 39 articles included information about 34 different quality indicators. Table 2 <ImgLink imgNo="2" imgType="table"/> shows the list of the quality indicators along with a reference to the indicators listed in <TextGroup><PlainText>Table 1 </PlainText></TextGroup><ImgLink imgNo="1" imgType="table"/>. Ten indicators from the guideline could not be attached to any of the indicators mentioned in the literature (20&#37; from 51 indicators). Four of those ten indicators had been introduced by representatives of the cancer registry, two by representatives of SHIP, and one by representatives of the data repository from OpEN.SC. </Pgraph><Pgraph>Thirteen indicators mentioned in the literature were not addressed in the guideline (38&#37; from 34 indicators): accessibility, appropriate amount of data, availability, believability, contextualization, granularity, inaccuracy, policy relevance, predictive value, relevancy, responsiveness of data items, spatial stability, and timeliness. Combining both sets, 64 indicators were available.</Pgraph></TextBlock> <TextBlock linked="yes" name="Discussion"> <MainHeadline>Discussion</MainHeadline><Pgraph>The second version of the TMF guideline for the management of data quality in cohorts and registries offers <TextGroup><PlainText>51 i</PlainText></TextGroup>ndicators for the assessment of data quality. A systematic literature review revealed 34 quality indicators used in 31 articles. Three indicators were mentioned ten times or more: accuracy, comprehensiveness and correctness. The mapping between both sets of indicators failed for ten of the TMF indicators (1031, 1032, 1033, 1034, 1037, 1038, 1040, 1041, 1042, and 1047). Nine out of those ten indicators are defined in the category organization. There seems to be a lack of understanding in the literature concerning the importance of measures related to the organization of cohorts and registries. Furthermore, seven of the missing indicators in the category organization were applied in data management of real data sets. Indicator TMF-1047 &#8220;Compliance with operating procedures&#8221; is not only missing in the reviewed literature, but also in the introduced conceptual frameworks <TextLink reference="7"></TextLink>, <TextLink reference="9"></TextLink>, <TextLink reference="10"></TextLink>. This neglects the requirements of empirical research to be compliant with predefined procedures, e.g. the timeline of follow-ups defined in the study protocol. One might assume that the elaboration of data quality focused in the past on the data itself neglecting the importance of process-related issues to some extent. </Pgraph><Pgraph>Thirteen out of the 34 indicators from the literature remained without a corresponding TMF indicator. Twelve were less frequently mentioned in the literature with only one or two citations. Timeliness was mentioned six times. <TextGroup><PlainText>Time</PlainText></TextGroup>liness is an important issue particularly in diagnosis and treatment as well as for reimbursement. Timeliness is of minor importance for research purposes. However, to offer a comprehensive set of quality indicators, timeliness should be added. The other twelve are to some extend overlapping with other indicators proposed in the literature. For example, there is an unclear relationship between accuracy, believability, correctness, and validity. Proposals had been made in the literature to clarify the definitions <TextLink reference="54"></TextLink>. However, the differentiation is still unclear. Contextualization, policy relevance, responsiveness of data items, and spatial stability extended the list of TMF indicators as well as the health care related conceptual frameworks mentioned before. These indicators could be assigned to the dimension contextual data quality defined by Wang and Strong. They already stated that contextual data quality &#8220;was not explicitly recognized in the data quality literature&#8221; <TextLink reference="8"></TextLink>. Possibly, this conflicts with the paradigm of empirical research and health care, to define the tasks first and collect the required data second. Then, the usefulness of the data is guaranteed by the predefined usage. However, in view of an increasing use of already existing data, contextual data quality might receive a greater importance in the future <TextLink reference="55"></TextLink>, <TextLink reference="56"></TextLink>, <TextLink reference="57"></TextLink>.</Pgraph><Pgraph>Some shortcomings have to be mentioned concerning the list of quality indicators in Table 1 <ImgLink imgNo="1" imgType="table"/>. The granularity of the indicator denominations and definitions varies, having broad measures as concordance on the one hand and particular measures as the rate of Death Certificate Only cases (DCO rate) on the other hand. The hierarchical organization is an attempt to address those differences. However, this solution is still suboptimal. Terms like &#8220;data element&#8221; and &#8220;value&#8221; are not always precise enough to represent the content of the indicator by its denomination. Therefore, the structured description offered in the long version of the guideline is essential to understand the meaning of an indicator.</Pgraph></TextBlock> <TextBlock linked="yes" name="Conclusions"> <MainHeadline>Conclusions</MainHeadline><Pgraph>The list of indicators for data quality derived in the presented project (cf. Table 1 <ImgLink imgNo="1" imgType="table"/>) covers many of the concepts used in the literature. It combines different perspectives, all relevant for data quality, </Pgraph><Pgraph><OrderedList><ListItem level="1" levelPosition="1" numString="1.">the perspective of data management responsible for data collection and data control, </ListItem><ListItem level="1" levelPosition="2" numString="2.">the perspective of data users, being unable to influence the process of data acquisition, and </ListItem><ListItem level="1" levelPosition="3" numString="3.">the perspective of process owners, defining the host projects and studies. </ListItem></OrderedList></Pgraph><Pgraph>Therefore, that list could be the starting point for a harmonization of indicators of data quality urgently needed noticing the variety and sometimes incompatibility of the measures mentioned in the literature. It will be a next step to offer a synthesis of both lists presented here along with precise definitions. This could be a valuable mission for standardization organizations that deal with data in health care and health care research. This research should take into account proposals for a formal definition of indicators <TextLink reference="58"></TextLink>. Formal definitions would enable an automatic application of indicators to data sets, for example offering a syntax for statistical software based on standards for metadata <TextLink reference="59"></TextLink>.</Pgraph></TextBlock> <TextBlock linked="yes" name="Notes"> <MainHeadline>Notes</MainHeadline><SubHeadline>Funding</SubHeadline><Pgraph>The project was funded under contract V020&#95;05 by the TMF &#8211; Technology, Methods, and Infrastructure for Networked Medical Research (cf. <Hyperlink href="http:&#47;&#47;www.tmf-ev.de&#47;">http:&#47;&#47;www.tmf-ev.de&#47;</Hyperlink>) and contract STA 454&#47;17-1 by the Deutsche Forschungsgemeinschaft (DFG).</Pgraph><SubHeadline>Competing interests</SubHeadline><Pgraph>The authors declare that they have no competing interests. </Pgraph></TextBlock> <References linked="yes"> <Reference refNo="1"> <RefAuthor>Arts DG</RefAuthor> <RefAuthor>De Keizer NF</RefAuthor> <RefAuthor>Scheffer GJ</RefAuthor> <RefTitle>Defining and improving data quality in medical registries: a literature review, case study, and generic framework</RefTitle> <RefYear>2002</RefYear> <RefJournal>J Am Med Inform Assoc</RefJournal> <RefPage>600-11</RefPage> <RefTotal>Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inform Assoc. 2002 Nov-Dec;9(6):600-11. DOI: 10.1197&#47;jamia.M1087</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1197&#47;jamia.M1087</RefLink> </Reference> <Reference refNo="2"> <RefAuthor>Nonnemacher M</RefAuthor> <RefAuthor>Weiland D</RefAuthor> <RefAuthor>Neuh&#228;user M</RefAuthor> <RefAuthor>Stausberg J</RefAuthor> <RefTitle>Adaptive management of data quality in cohort studies and registers: proposal for a guideline</RefTitle> <RefYear>2007</RefYear> <RefJournal>Acta Inform Med</RefJournal> <RefPage>225-30</RefPage> <RefTotal>Nonnemacher M, Weiland D, Neuh&#228;user M, Stausberg J. Adaptive management of data quality in cohort studies and registers: proposal for a guideline. Acta Inform Med. 2007;15:225-30.</RefTotal> </Reference> <Reference refNo="3"> <RefAuthor>Coorevits P</RefAuthor> <RefAuthor>Sundgren M</RefAuthor> <RefAuthor>Klein GO</RefAuthor> <RefAuthor>Bahr A</RefAuthor> <RefAuthor>Claerhout B</RefAuthor> <RefAuthor>Daniel C</RefAuthor> <RefAuthor>Dugas M</RefAuthor> <RefAuthor>Dupont D</RefAuthor> <RefAuthor>Schmidt A</RefAuthor> <RefAuthor>Singleton P</RefAuthor> <RefAuthor>De Moor G</RefAuthor> <RefAuthor>Kalra D</RefAuthor> <RefTitle>Electronic health records: new opportunities for clinical research</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Intern Med</RefJournal> <RefPage>547-60</RefPage> <RefTotal>Coorevits P, Sundgren M, Klein GO, Bahr A, Claerhout B, Daniel C, Dugas M, Dupont D, Schmidt A, Singleton P, De Moor G, Kalra D. Electronic health records: new opportunities for clinical research. J Intern Med. 2013 Dec;274(6):547-60. DOI: 10.1111&#47;joim.12119</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1111&#47;joim.12119</RefLink> </Reference> <Reference refNo="4"> <RefAuthor>Malin JL</RefAuthor> <RefAuthor>Keating NL</RefAuthor> <RefTitle>The cost-quality trade-off: need for data quality standards for studies that impact clinical practice and health policy</RefTitle> <RefYear>2005</RefYear> <RefJournal>J Clin Oncol</RefJournal> <RefPage>4581-4</RefPage> <RefTotal>Malin JL, Keating NL. The cost-quality trade-off: need for data quality standards for studies that impact clinical practice and health policy. J Clin Oncol. 2005 Jul;23(21):4581-4. DOI: 10.1200&#47;JCO.2005.01.912</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1200&#47;JCO.2005.01.912</RefLink> </Reference> <Reference refNo="5"> <RefAuthor>Joint Commission on Accreditation of Healthcare Organisations (JCAHO)</RefAuthor> <RefTitle></RefTitle> <RefYear>1990</RefYear> <RefBookTitle>Primer on indicator development and application. Measuring Quality in Health Care</RefBookTitle> <RefPage></RefPage> <RefTotal>Joint Commission on Accreditation of Healthcare Organisations (JCAHO). Primer on indicator development and application. Measuring Quality in Health Care. Oakbrook Terrace: JCAHO; 1990.</RefTotal> </Reference> <Reference refNo="6"> <RefAuthor>Brooke EM</RefAuthor> <RefTitle></RefTitle> <RefYear>1974</RefYear> <RefBookTitle>The current and future use of registries in health information systems</RefBookTitle> <RefPage></RefPage> <RefTotal>Brooke EM. The current and future use of registries in health information systems. Geneva: World Health Organization; 1974.</RefTotal> </Reference> <Reference refNo="7"> <RefAuthor>Botsis T</RefAuthor> <RefAuthor>Hartvigsen G</RefAuthor> <RefAuthor>Chen F</RefAuthor> <RefAuthor>Weng C</RefAuthor> <RefTitle>Secondary Use of EHR: Data Quality Issues and Informatics Opportunities</RefTitle> <RefYear>2010</RefYear> <RefJournal>Summit Transl Bioinform</RefJournal> <RefPage>1-5</RefPage> <RefTotal>Botsis T, Hartvigsen G, Chen F, Weng C. Secondary Use of EHR: Data Quality Issues and Informatics Opportunities. Summit Transl Bioinform. 2010 Mar 1;2010:1-5.</RefTotal> </Reference> <Reference refNo="8"> <RefAuthor>Wang RY</RefAuthor> <RefAuthor>Strong DM</RefAuthor> <RefTitle>Beyond accuracy: what data quality means to data consumers</RefTitle> <RefYear>1996</RefYear> <RefJournal>J Manag Inf Syst</RefJournal> <RefPage>5-33</RefPage> <RefTotal>Wang RY, Strong DM. Beyond accuracy: what data quality means to data consumers. J Manag Inf Syst. 1996;12:5-33. DOI: 10.1080&#47;07421222.1996.11518099</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1080&#47;07421222.1996.11518099</RefLink> </Reference> <Reference refNo="9"> <RefAuthor>Weiskopf NG</RefAuthor> <RefAuthor>Weng C</RefAuthor> <RefTitle>Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Am Med Inform Assoc</RefJournal> <RefPage>144-51</RefPage> <RefTotal>Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013 Jan;20(1):144-51. DOI: 10.1136&#47;amiajnl-2011-000681</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1136&#47;amiajnl-2011-000681</RefLink> </Reference> <Reference refNo="10"> <RefAuthor>Kahn MG</RefAuthor> <RefAuthor>Callahan TJ</RefAuthor> <RefAuthor>Barnard J</RefAuthor> <RefAuthor>Bauck AE</RefAuthor> <RefAuthor>Brown J</RefAuthor> <RefAuthor>Davidson BN</RefAuthor> <RefAuthor>Estiri H</RefAuthor> <RefAuthor>Goerg C</RefAuthor> <RefAuthor>Holve E</RefAuthor> <RefAuthor>Johnson SG</RefAuthor> <RefAuthor>Liaw ST</RefAuthor> <RefAuthor>Hamilton-Lopez M</RefAuthor> <RefAuthor>Meeker D</RefAuthor> <RefAuthor>Ong TC</RefAuthor> <RefAuthor>Ryan P</RefAuthor> <RefAuthor>Shang N</RefAuthor> <RefAuthor>Weiskopf NG</RefAuthor> <RefAuthor>Weng C</RefAuthor> <RefAuthor>Zozus MN</RefAuthor> <RefAuthor>Schilling L</RefAuthor> <RefTitle>A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data</RefTitle> <RefYear>2016</RefYear> <RefJournal>EGEMS (Wash DC)</RefJournal> <RefPage>1244</RefPage> <RefTotal>Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, Estiri H, Goerg C, Holve E, Johnson SG, Liaw ST, Hamilton-Lopez M, Meeker D, Ong TC, Ryan P, Shang N, Weiskopf NG, Weng C, Zozus MN, Schilling L. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. EGEMS (Wash DC). 2016 Sep 11;4(1):1244. DOI: 10.13063&#47;2327-9214.1244</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.13063&#47;2327-9214.1244</RefLink> </Reference> <Reference refNo="11"> <RefAuthor>Nonnemacher M</RefAuthor> <RefAuthor>Weiland D</RefAuthor> <RefAuthor>Stausberg J</RefAuthor> <RefTitle></RefTitle> <RefYear>2007</RefYear> <RefBookTitle>Datenqualit&#228;t in der medizinischen Forschung. Leitlinie zum adaptiven Management von Datenqualit&#228;t in Kohortenstudien und Registern</RefBookTitle> <RefPage></RefPage> <RefTotal>Nonnemacher M, Weiland D, Stausberg J. Datenqualit&#228;t in der medizinischen Forschung. Leitlinie zum adaptiven Management von Datenqualit&#228;t in Kohortenstudien und Registern. Berlin: Medizinisch Wissenschaftliche Verlagsgesellschaft; 2007.</RefTotal> </Reference> <Reference refNo="12"> <RefAuthor>Donabedian A</RefAuthor> <RefTitle>Evaluating the quality of medical care</RefTitle> <RefYear>1966</RefYear> <RefJournal>Milbank Mem Fund Q</RefJournal> <RefPage>166-206</RefPage> <RefTotal>Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966 Jul;44(3):166-206.</RefTotal> </Reference> <Reference refNo="13"> <RefAuthor>Deming WE</RefAuthor> <RefTitle></RefTitle> <RefYear>1982</RefYear> <RefBookTitle>Out of the crisis</RefBookTitle> <RefPage></RefPage> <RefTotal>Deming WE. Out of the crisis. Cambridge: Cambridge University Press; 1982.</RefTotal> </Reference> <Reference refNo="14"> <RefAuthor>Stausberg J</RefAuthor> <RefAuthor>Pritzkuleit R</RefAuthor> <RefAuthor>Schmidt CO</RefAuthor> <RefAuthor>Schrader T</RefAuthor> <RefAuthor>Nonnemacher M</RefAuthor> <RefTitle>Indicators of data quality: revision of a guideline for networked medical research</RefTitle> <RefYear>2012</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>711-5</RefPage> <RefTotal>Stausberg J, Pritzkuleit R, Schmidt CO, Schrader T, Nonnemacher M. Indicators of data quality: revision of a guideline for networked medical research. Stud Health Technol Inform. 2012;180:711-5.</RefTotal> </Reference> <Reference refNo="15"> <RefAuthor>Ziegler A</RefAuthor> <RefAuthor>K&#246;nig IR</RefAuthor> <RefTitle>Leitlinien fur Forschungsberichte: Deutschsprachige &#220;bersetzungen von CONSORT 2010, PRISMA und STARD.</RefTitle> <RefYear>2011</RefYear> <RefJournal>Dtsch Med Wochenschr</RefJournal> <RefPage>357-8</RefPage> <RefTotal>Ziegler A, K&#246;nig IR. Leitlinien fur Forschungsberichte: Deutschsprachige &#220;bersetzungen von CONSORT 2010, PRISMA und STARD. &#91;Guidelines for research reports: German translation of CONSORT 2010, PRISMA and STARD&#93;. Dtsch Med Wochenschr. 2011 Feb;136(8):357-8. DOI: 10.1055&#47;s-0031-1272535</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1055&#47;s-0031-1272535</RefLink> </Reference> <Reference refNo="16"> <RefAuthor>Nonnemacher M</RefAuthor> <RefAuthor>Nasseh D</RefAuthor> <RefAuthor>Stausberg J</RefAuthor> <RefTitle></RefTitle> <RefYear>2014</RefYear> <RefBookTitle>Datenqualit&#228;t in der medizinischen Forschung</RefBookTitle> <RefPage></RefPage> <RefTotal>Nonnemacher M, Nasseh D, Stausberg J. Datenqualit&#228;t in der medizinischen Forschung. 2., aktual. u. erw. Aufl. Berlin: Medizinisch Wissenschaftliche Verlagsgesellschaft; 2014.</RefTotal> </Reference> <Reference refNo="17"> <RefAuthor>Baigent C</RefAuthor> <RefAuthor>Harrell FE</RefAuthor> <RefAuthor>Buyse M</RefAuthor> <RefAuthor>Emberson JR</RefAuthor> <RefAuthor>Altman DG</RefAuthor> <RefTitle>Ensuring trial validity by data quality assurance and diversification of monitoring methods</RefTitle> <RefYear>2008</RefYear> <RefJournal>Clin Trials</RefJournal> <RefPage>49-55</RefPage> <RefTotal>Baigent C, Harrell FE, Buyse M, Emberson JR, Altman DG. Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin Trials. 2008;5(1):49-55. DOI: 10.1177&#47;1740774507087554</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1177&#47;1740774507087554</RefLink> </Reference> <Reference refNo="18"> <RefAuthor>Berner ES</RefAuthor> <RefAuthor>Kasiraman RK</RefAuthor> <RefAuthor>Yu F</RefAuthor> <RefAuthor>Ray MN</RefAuthor> <RefAuthor>Houston TK</RefAuthor> <RefTitle>Data quality in the outpatient setting: impact on clinical decision support systems</RefTitle> <RefYear>2005</RefYear> <RefJournal>AMIA Annu Symp Proc</RefJournal> <RefPage>41-5</RefPage> <RefTotal>Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annu Symp Proc. 2005:41-5.</RefTotal> </Reference> <Reference refNo="19"> <RefAuthor>Bray F</RefAuthor> <RefAuthor>Parkin DM</RefAuthor> <RefTitle>Evaluation of data quality in the cancer registry: principles and methods. Part I: comparability, validity and timeliness</RefTitle> <RefYear>2009</RefYear> <RefJournal>Eur J Cancer</RefJournal> <RefPage>747-55</RefPage> <RefTotal>Bray F, Parkin DM. Evaluation of data quality in the cancer registry: principles and methods. Part I: comparability, validity and timeliness. Eur J Cancer. 2009 Mar;45(5):747-55. DOI: 10.1016&#47;j.ejca.2008.11.032</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.ejca.2008.11.032</RefLink> </Reference> <Reference refNo="20"> <RefAuthor>Brender JD</RefAuthor> <RefAuthor>Suarez L</RefAuthor> <RefAuthor>Langlois PH</RefAuthor> <RefTitle>Validity of parental work information on the birth certificate</RefTitle> <RefYear>2008</RefYear> <RefJournal>BMC Public Health</RefJournal> <RefPage>95</RefPage> <RefTotal>Brender JD, Suarez L, Langlois PH. Validity of parental work information on the birth certificate. BMC Public Health. 2008 Mar;8:95. DOI: 10.1186&#47;1471-2458-8-95</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1186&#47;1471-2458-8-95</RefLink> </Reference> <Reference refNo="21"> <RefAuthor>Bronnert J</RefAuthor> <RefAuthor>Clark JS</RefAuthor> <RefAuthor>Cassidy BS</RefAuthor> <RefAuthor>Fenton S</RefAuthor> <RefAuthor>Hyde L</RefAuthor> <RefAuthor>Kallem C</RefAuthor> <RefAuthor>Watzlaf V</RefAuthor> <RefTitle>Data quality management model (updated)</RefTitle> <RefYear>2012</RefYear> <RefJournal>J AHIMA</RefJournal> <RefPage>62-71</RefPage> <RefTotal>Bronnert J, Clark JS, Cassidy BS, Fenton S, Hyde L, Kallem C, Watzlaf V. Data quality management model (updated). J AHIMA. 2012 Jul;83(7):62-71.</RefTotal> </Reference> <Reference refNo="22"> <RefAuthor>Brouwer HJ</RefAuthor> <RefAuthor>Bindels PJ</RefAuthor> <RefAuthor>Weert HC</RefAuthor> <RefTitle>Data quality improvement in general practice</RefTitle> <RefYear>2006</RefYear> <RefJournal>Fam Pract</RefJournal> <RefPage>529-36</RefPage> <RefTotal>Brouwer HJ, Bindels PJ, Weert HC. Data quality improvement in general practice. Fam Pract. 2006 Oct;23(5):529-36. DOI: 10.1093&#47;fampra&#47;cml040</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1093&#47;fampra&#47;cml040</RefLink> </Reference> <Reference refNo="23"> <RefAuthor>Chiba Y</RefAuthor> <RefAuthor>Oguttu MA</RefAuthor> <RefAuthor>Nakayama T</RefAuthor> <RefTitle>Quantitative and qualitative verification of data quality in the childbirth registers of two rural district hospitals in Western Kenya</RefTitle> <RefYear>2012</RefYear> <RefJournal>Midwifery</RefJournal> <RefPage>329-39</RefPage> <RefTotal>Chiba Y, Oguttu MA, Nakayama T. Quantitative and qualitative verification of data quality in the childbirth registers of two rural district hospitals in Western Kenya. Midwifery. 2012 Jun;28(3):329-39. DOI: 10.1016&#47;j.midw.2011.05.005</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.midw.2011.05.005</RefLink> </Reference> <Reference refNo="24"> <RefAuthor>Choquet R</RefAuthor> <RefAuthor>Qouiyd S</RefAuthor> <RefAuthor>Ouagne D</RefAuthor> <RefAuthor>Pasche E</RefAuthor> <RefAuthor>Daniel C</RefAuthor> <RefAuthor>Boussa&#239;d O</RefAuthor> <RefAuthor>Jaulent MC</RefAuthor> <RefTitle>The Information Quality Triangle: a methodology to assess clinical information quality</RefTitle> <RefYear>2010</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>699-703</RefPage> <RefTotal>Choquet R, Qouiyd S, Ouagne D, Pasche E, Daniel C, Boussa&#239;d O, Jaulent MC. The Information Quality Triangle: a methodology to assess clinical information quality. Stud Health Technol Inform. 2010;160(Pt 1):699-703.</RefTotal> </Reference> <Reference refNo="25"> <RefAuthor>Couchoud C</RefAuthor> <RefAuthor>Lassalle M</RefAuthor> <RefAuthor>Cornet R</RefAuthor> <RefAuthor>Jager KJ</RefAuthor> <RefTitle>Renal replacement therapy registries &#8211; time for a structured data quality evaluation programme</RefTitle> <RefYear>2013</RefYear> <RefJournal>Nephrol Dial Transplant</RefJournal> <RefPage>2215-20</RefPage> <RefTotal>Couchoud C, Lassalle M, Cornet R, Jager KJ. Renal replacement therapy registries &#8211; time for a structured data quality evaluation programme. Nephrol Dial Transplant. 2013 Sep;28(9):2215-20. DOI: 10.1093&#47;ndt&#47;gft004</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1093&#47;ndt&#47;gft004</RefLink> </Reference> <Reference refNo="26"> <RefAuthor>De S</RefAuthor> <RefTitle>Hybrid approaches to clinical trial monitoring: Practical alternatives to 100&#37; source data verification</RefTitle> <RefYear>2011</RefYear> <RefJournal>Perspect Clin Res</RefJournal> <RefPage>100-4</RefPage> <RefTotal>De S. Hybrid approaches to clinical trial monitoring: Practical alternatives to 100&#37; source data verification. Perspect Clin Res. 2011 Jul;2(3):100-4. DOI: 10.4103&#47;2229-3485.83226</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.4103&#47;2229-3485.83226</RefLink> </Reference> <Reference refNo="27"> <RefAuthor>Duda SN</RefAuthor> <RefAuthor>Shepherd BE</RefAuthor> <RefAuthor>Gadd CS</RefAuthor> <RefAuthor>Masys DR</RefAuthor> <RefAuthor>McGowan CC</RefAuthor> <RefTitle>Measuring the quality of observational study data in an international HIV research network</RefTitle> <RefYear>2012</RefYear> <RefJournal>PLoS ONE</RefJournal> <RefPage>e33908</RefPage> <RefTotal>Duda SN, Shepherd BE, Gadd CS, Masys DR, McGowan CC. Measuring the quality of observational study data in an international HIV research network. PLoS ONE. 2012;7(4):e33908. DOI: 10.1371&#47;journal.pone.0033908</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1371&#47;journal.pone.0033908</RefLink> </Reference> <Reference refNo="28"> <RefAuthor>Dyck MJ</RefAuthor> <RefAuthor>Culp K</RefAuthor> <RefAuthor>Cacchione PZ</RefAuthor> <RefTitle>Data quality strategies in cohort studies: lessons from a study on delirium in nursing home elders</RefTitle> <RefYear>2007</RefYear> <RefJournal>Appl Nurs Res</RefJournal> <RefPage>39-43</RefPage> <RefTotal>Dyck MJ, Culp K, Cacchione PZ. Data quality strategies in cohort studies: lessons from a study on delirium in nursing home elders. Appl Nurs Res. 2007 Feb;20(1):39-43. DOI: 10.1016&#47;j.apnr.2006.01.004</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.apnr.2006.01.004</RefLink> </Reference> <Reference refNo="29"> <RefAuthor>Fran&#231;a E</RefAuthor> <RefAuthor>de Abreu DX</RefAuthor> <RefAuthor>Rao C</RefAuthor> <RefAuthor>Lopez AD</RefAuthor> <RefTitle>Evaluation of cause-of-death statistics for Brazil, 2002-2004</RefTitle> <RefYear>2008</RefYear> <RefJournal>Int J Epidemiol</RefJournal> <RefPage>891-901</RefPage> <RefTotal>Fran&#231;a E, de Abreu DX, Rao C, Lopez AD. Evaluation of cause-of-death statistics for Brazil, 2002-2004. Int J Epidemiol. 2008 Aug;37(4):891-901. DOI: 10.1093&#47;ije&#47;dyn121</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1093&#47;ije&#47;dyn121</RefLink> </Reference> <Reference refNo="30"> <RefAuthor>Kahn MG</RefAuthor> <RefAuthor>Raebel MA</RefAuthor> <RefAuthor>Glanz JM</RefAuthor> <RefAuthor>Riedlinger K</RefAuthor> <RefAuthor>Steiner JF</RefAuthor> <RefTitle>A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research</RefTitle> <RefYear>2012</RefYear> <RefJournal>Med Care</RefJournal> <RefPage>S21-9</RefPage> <RefTotal>Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med Care. 2012 Jul;50 Suppl:S21-9. DOI: 10.1097&#47;MLR.0b013e318257dd67</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1097&#47;MLR.0b013e318257dd67</RefLink> </Reference> <Reference refNo="31"> <RefAuthor>Krzych LJ</RefAuthor> <RefAuthor>Lees B</RefAuthor> <RefAuthor>Nugara F</RefAuthor> <RefAuthor>Banya W</RefAuthor> <RefAuthor>Bochenek A</RefAuthor> <RefAuthor>Cook J</RefAuthor> <RefAuthor>Taggart D</RefAuthor> <RefAuthor>Flather MD</RefAuthor> <RefTitle>Assessment of data quality in an international multi-centre randomised trial of coronary artery surgery</RefTitle> <RefYear>2011</RefYear> <RefJournal>Trials</RefJournal> <RefPage>212</RefPage> <RefTotal>Krzych LJ, Lees B, Nugara F, Banya W, Bochenek A, Cook J, Taggart D, Flather MD. Assessment of data quality in an international multi-centre randomised trial of coronary artery surgery. Trials. 2011 Sep;12:212. DOI: 10.1186&#47;1745-6215-12-212</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1186&#47;1745-6215-12-212</RefLink> </Reference> <Reference refNo="32"> <RefAuthor>Larsen IK</RefAuthor> <RefAuthor>Sm&#229;stuen M</RefAuthor> <RefAuthor>Johannesen TB</RefAuthor> <RefAuthor>Langmark F</RefAuthor> <RefAuthor>Parkin DM</RefAuthor> <RefAuthor>Bray F</RefAuthor> <RefAuthor>M&#248;ller B</RefAuthor> <RefTitle>Data quality at the Cancer Registry of Norway: an overview of comparability, completeness, validity and timeliness</RefTitle> <RefYear>2009</RefYear> <RefJournal>Eur J Cancer</RefJournal> <RefPage>1218-31</RefPage> <RefTotal>Larsen IK, Sm&#229;stuen M, Johannesen TB, Langmark F, Parkin DM, Bray F, M&#248;ller B. Data quality at the Cancer Registry of Norway: an overview of comparability, completeness, validity and timeliness. Eur J Cancer. 2009 May;45(7):1218-31. DOI: 10.1016&#47;j.ejca.2008.10.037</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.ejca.2008.10.037</RefLink> </Reference> <Reference refNo="33"> <RefAuthor>Loane M</RefAuthor> <RefAuthor>Dolk H</RefAuthor> <RefAuthor>Garne E</RefAuthor> <RefAuthor>Greenlees R</RefAuthor> <RefAuthor> EUROCAT Working Group</RefAuthor> <RefTitle>Paper 3: EUROCAT data quality indicators for population-based registries of congenital anomalies</RefTitle> <RefYear>2011</RefYear> <RefJournal>Birth Defects Res Part A Clin Mol Teratol</RefJournal> <RefPage>S23-30</RefPage> <RefTotal>Loane M, Dolk H, Garne E, Greenlees R; EUROCAT Working Group. Paper 3: EUROCAT data quality indicators for population-based registries of congenital anomalies. Birth Defects Res Part A Clin Mol Teratol. 2011 Mar;91 Suppl 1:S23-30. DOI: 10.1002&#47;bdra.20779</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1002&#47;bdra.20779</RefLink> </Reference> <Reference refNo="34"> <RefAuthor>Macefield RC</RefAuthor> <RefAuthor>Beswick AD</RefAuthor> <RefAuthor>Blazeby JM</RefAuthor> <RefAuthor>Lane JA</RefAuthor> <RefTitle>A systematic review of on-site monitoring methods for health-care randomised controlled trials</RefTitle> <RefYear>2013</RefYear> <RefJournal>Clin Trials</RefJournal> <RefPage>104-24</RefPage> <RefTotal>Macefield RC, Beswick AD, Blazeby JM, Lane JA. A systematic review of on-site monitoring methods for health-care randomised controlled trials. Clin Trials. 2013 Feb;10(1):104-24. DOI: 10.1177&#47;1740774512467405</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1177&#47;1740774512467405</RefLink> </Reference> <Reference refNo="35"> <RefAuthor>Maruszewski B</RefAuthor> <RefAuthor>Lacour-Gayet F</RefAuthor> <RefAuthor>Monro JL</RefAuthor> <RefAuthor>Keogh BE</RefAuthor> <RefAuthor>Tobota Z</RefAuthor> <RefAuthor>Kansy A</RefAuthor> <RefTitle>An attempt at data verification in the EACTS Congenital Database</RefTitle> <RefYear>2005</RefYear> <RefJournal>Eur J Cardiothorac Surg</RefJournal> <RefPage>400-4; discussion 405-6</RefPage> <RefTotal>Maruszewski B, Lacour-Gayet F, Monro JL, Keogh BE, Tobota Z, Kansy A. An attempt at data verification in the EACTS Congenital Database. Eur J Cardiothorac Surg. 2005 Sep;28(3):400-4; discussion 405-6. DOI: 10.1016&#47;j.ejcts.2005.03.051</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.ejcts.2005.03.051</RefLink> </Reference> <Reference refNo="36"> <RefAuthor>McKenzie K</RefAuthor> <RefAuthor>Walker S</RefAuthor> <RefAuthor>Besenyei A</RefAuthor> <RefAuthor>Aitken LM</RefAuthor> <RefAuthor>Allison B</RefAuthor> <RefTitle>Assessing the concordance of trauma registry data and hospital records</RefTitle> <RefYear>2005</RefYear> <RefJournal>Health Inf Manag</RefJournal> <RefPage>3-7</RefPage> <RefTotal>McKenzie K, Walker S, Besenyei A, Aitken LM, Allison B. Assessing the concordance of trauma registry data and hospital records. Health Inf Manag. 2005;34(1):3-7. DOI: 10.1177&#47;183335830503400103</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1177&#47;183335830503400103</RefLink> </Reference> <Reference refNo="37"> <RefAuthor>Messenger JC</RefAuthor> <RefAuthor>Ho KK</RefAuthor> <RefAuthor>Young CH</RefAuthor> <RefAuthor>Slattery LE</RefAuthor> <RefAuthor>Draoui JC</RefAuthor> <RefAuthor>Curtis JP</RefAuthor> <RefAuthor>Dehmer GJ</RefAuthor> <RefAuthor>Grover FL</RefAuthor> <RefAuthor>Mirro MJ</RefAuthor> <RefAuthor>Reynolds MR</RefAuthor> <RefAuthor>Rokos IC</RefAuthor> <RefAuthor>Spertus JA</RefAuthor> <RefAuthor>Wang TY</RefAuthor> <RefAuthor>Winston SA</RefAuthor> <RefAuthor>Rumsfeld JS</RefAuthor> <RefAuthor>Masoudi FA</RefAuthor> <RefAuthor> NCDR Science and Quality Oversight Committee Data Quality Workgroup</RefAuthor> <RefTitle>The National Cardiovascular Data Registry (NCDR) Data Quality Brief: the NCDR Data Quality Program in 2012</RefTitle> <RefYear>2012</RefYear> <RefJournal>J Am Coll Cardiol</RefJournal> <RefPage>1484-8</RefPage> <RefTotal>Messenger JC, Ho KK, Young CH, Slattery LE, Draoui JC, Curtis JP, Dehmer GJ, Grover FL, Mirro MJ, Reynolds MR, Rokos IC, Spertus JA, Wang TY, Winston SA, Rumsfeld JS, Masoudi FA; NCDR Science and Quality Oversight Committee Data Quality Workgroup. The National Cardiovascular Data Registry (NCDR) Data Quality Brief: the NCDR Data Quality Program in 2012. J Am Coll Cardiol. 2012 Oct;60(16):1484-8. DOI: 10.1016&#47;j.jacc.2012.07.020</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.jacc.2012.07.020</RefLink> </Reference> <Reference refNo="38"> <RefAuthor>Mphatswe W</RefAuthor> <RefAuthor>Mate KS</RefAuthor> <RefAuthor>Bennett B</RefAuthor> <RefAuthor>Ngidi H</RefAuthor> <RefAuthor>Reddy J</RefAuthor> <RefAuthor>Barker PM</RefAuthor> <RefAuthor>Rollins N</RefAuthor> <RefTitle>Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa</RefTitle> <RefYear>2012</RefYear> <RefJournal>Bull World Health Organ</RefJournal> <RefPage>176-82</RefPage> <RefTotal>Mphatswe W, Mate KS, Bennett B, Ngidi H, Reddy J, Barker PM, Rollins N. Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa. Bull World Health Organ. 2012 Mar;90(3):176-82. DOI: 10.2471&#47;BLT.11.092759</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.2471&#47;BLT.11.092759</RefLink> </Reference> <Reference refNo="39"> <RefAuthor>Nahm ML</RefAuthor> <RefAuthor>Pieper CF</RefAuthor> <RefAuthor>Cunningham MM</RefAuthor> <RefTitle>Quantifying data quality for clinical trials using electronic data capture</RefTitle> <RefYear>2008</RefYear> <RefJournal>PLoS ONE</RefJournal> <RefPage>e3049</RefPage> <RefTotal>Nahm ML, Pieper CF, Cunningham MM. Quantifying data quality for clinical trials using electronic data capture. PLoS ONE. 2008 Aug;3(8):e3049. DOI: 10.1371&#47;journal.pone.0003049</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1371&#47;journal.pone.0003049</RefLink> </Reference> <Reference refNo="40"> <RefAuthor>S&#225;ez C</RefAuthor> <RefAuthor>Mart&#237;nez-Miranda J</RefAuthor> <RefAuthor>Robles M</RefAuthor> <RefAuthor>Garc&#237;a-G&#243;mez JM</RefAuthor> <RefTitle>Organizing data quality assessment of shifting biomedical data</RefTitle> <RefYear>2012</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>721-5</RefPage> <RefTotal>S&#225;ez C, Mart&#237;nez-Miranda J, Robles M, Garc&#237;a-G&#243;mez JM. Organizing data quality assessment of shifting biomedical data. Stud Health Technol Inform. 2012;180:721-5.</RefTotal> </Reference> <Reference refNo="41"> <RefAuthor>Salati M</RefAuthor> <RefAuthor>Brunelli A</RefAuthor> <RefAuthor>Dahan M</RefAuthor> <RefAuthor>Rocco G</RefAuthor> <RefAuthor>Van Raemdonck DE</RefAuthor> <RefAuthor>Varela G</RefAuthor> <RefAuthor> European Society of Thoracic Surgeons Database Committee</RefAuthor> <RefTitle>Task-independent metrics to assess the data quality of medical registries using the European Society of Thoracic Surgeons (ESTS) Database</RefTitle> <RefYear>2011</RefYear> <RefJournal>Eur J Cardiothorac Surg</RefJournal> <RefPage>91-8</RefPage> <RefTotal>Salati M, Brunelli A, Dahan M, Rocco G, Van Raemdonck DE, Varela G; European Society of Thoracic Surgeons Database Committee. Task-independent metrics to assess the data quality of medical registries using the European Society of Thoracic Surgeons (ESTS) Database. Eur J Cardiothorac Surg. 2011 Jul;40(1):91-8. DOI: 10.1016&#47;j.ejcts.2010.11.004</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.ejcts.2010.11.004</RefLink> </Reference> <Reference refNo="42"> <RefAuthor>Shabestari O</RefAuthor> <RefAuthor>Roudsari A</RefAuthor> <RefTitle>Challenges in data quality assurance for electronic health records</RefTitle> <RefYear>2013</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>37-41</RefPage> <RefTotal>Shabestari O, Roudsari A. Challenges in data quality assurance for electronic health records. Stud Health Technol Inform. 2013;183:37-41.</RefTotal> </Reference> <Reference refNo="43"> <RefAuthor>Sigurdardottir LG</RefAuthor> <RefAuthor>Jonasson JG</RefAuthor> <RefAuthor>Stefansdottir S</RefAuthor> <RefAuthor>Jonsdottir A</RefAuthor> <RefAuthor>Olafsdottir GH</RefAuthor> <RefAuthor>Olafsdottir EJ</RefAuthor> <RefAuthor>Tryggvadottir L</RefAuthor> <RefTitle>Data quality at the Icelandic Cancer Registry: comparability, validity, timeliness and completeness</RefTitle> <RefYear>2012</RefYear> <RefJournal>Acta Oncol</RefJournal> <RefPage>880-9</RefPage> <RefTotal>Sigurdardottir LG, Jonasson JG, Stefansdottir S, Jonsdottir A, Olafsdottir GH, Olafsdottir EJ, Tryggvadottir L. Data quality at the Icelandic Cancer Registry: comparability, validity, timeliness and completeness. Acta Oncol. 2012 Sep;51(7):880-9. DOI: 10.3109&#47;0284186X.2012.698751</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.3109&#47;0284186X.2012.698751</RefLink> </Reference> <Reference refNo="44"> <RefAuthor>Stevens W</RefAuthor> <RefAuthor>Stevens G</RefAuthor> <RefAuthor>Kolbe J</RefAuthor> <RefAuthor>Cox B</RefAuthor> <RefTitle>Comparison of New Zealand Cancer Registry data with an independent lung cancer audit</RefTitle> <RefYear>2008</RefYear> <RefJournal>N Z Med J</RefJournal> <RefPage>29-41</RefPage> <RefTotal>Stevens W, Stevens G, Kolbe J, Cox B. Comparison of New Zealand Cancer Registry data with an independent lung cancer audit. N Z Med J. 2008 Jun 20;121(1276):29-41. </RefTotal> </Reference> <Reference refNo="45"> <RefAuthor>Taggart J</RefAuthor> <RefAuthor>Liaw ST</RefAuthor> <RefAuthor>Dennis S</RefAuthor> <RefAuthor>Yu H</RefAuthor> <RefAuthor>Rahimi A</RefAuthor> <RefAuthor>Jalaludin B</RefAuthor> <RefAuthor>Harris M</RefAuthor> <RefTitle>The University of NSW electronic practice based research network: disease registers, data quality and utility</RefTitle> <RefYear>2012</RefYear> <RefJournal>Stud Health Technol Inform</RefJournal> <RefPage>219-27</RefPage> <RefTotal>Taggart J, Liaw ST, Dennis S, Yu H, Rahimi A, Jalaludin B, Harris M. The University of NSW electronic practice based research network: disease registers, data quality and utility. Stud Health Technol Inform. 2012;178:219-27. </RefTotal> </Reference> <Reference refNo="46"> <RefAuthor>Thoburn KK</RefAuthor> <RefAuthor>German RR</RefAuthor> <RefAuthor>Lewis M</RefAuthor> <RefAuthor>Nichols PJ</RefAuthor> <RefAuthor>Ahmed F</RefAuthor> <RefAuthor>Jackson-Thompson J</RefAuthor> <RefTitle>Case completeness and data accuracy in the Centers for Disease Control and Prevention&#8217;s National Program of Cancer Registries</RefTitle> <RefYear>2007</RefYear> <RefJournal>Cancer</RefJournal> <RefPage>1607-16</RefPage> <RefTotal>Thoburn KK, German RR, Lewis M, Nichols PJ, Ahmed F, Jackson-Thompson J. Case completeness and data accuracy in the Centers for Disease Control and Prevention&#8217;s National Program of Cancer Registries. Cancer. 2007 Apr;109(8):1607-16. DOI: 10.1002&#47;cncr.22566</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1002&#47;cncr.22566</RefLink> </Reference> <Reference refNo="47"> <RefAuthor>Tolonen H</RefAuthor> <RefAuthor>Dobson A</RefAuthor> <RefAuthor>Kulathinal S</RefAuthor> <RefAuthor> WHO MONICA Project</RefAuthor> <RefTitle>Assessing the quality of risk factor survey data: lessons from the WHO MONICA Project</RefTitle> <RefYear>2006</RefYear> <RefJournal>Eur J Cardiovasc Prev Rehabil</RefJournal> <RefPage>104-14</RefPage> <RefTotal>Tolonen H, Dobson A, Kulathinal S; WHO MONICA Project. Assessing the quality of risk factor survey data: lessons from the WHO MONICA Project. Eur J Cardiovasc Prev Rehabil. 2006 Feb;13(1):104-14. DOI: 10.1097&#47;00149831-200602000-00017</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1097&#47;00149831-200602000-00017</RefLink> </Reference> <Reference refNo="48"> <RefAuthor>Tuble SC</RefAuthor> <RefTitle>Perfusion Downunder Collaboration Database &#8211; data quality assurance: towards a high quality clinical database</RefTitle> <RefYear>2011</RefYear> <RefJournal>J Extra Corpor Technol</RefJournal> <RefPage>P44-51</RefPage> <RefTotal>Tuble SC. Perfusion Downunder Collaboration Database &#8211; data quality assurance: towards a high quality clinical database. J Extra Corpor Technol. 2011 Mar;43(1):P44-51.</RefTotal> </Reference> <Reference refNo="49"> <RefAuthor>Tudur Smith C</RefAuthor> <RefAuthor>Stocken DD</RefAuthor> <RefAuthor>Dunn J</RefAuthor> <RefAuthor>Cox T</RefAuthor> <RefAuthor>Ghaneh P</RefAuthor> <RefAuthor>Cunningham D</RefAuthor> <RefAuthor>Neoptolemos JP</RefAuthor> <RefTitle>The value of source data verification in a cancer clinical trial</RefTitle> <RefYear>2012</RefYear> <RefJournal>PLoS ONE</RefJournal> <RefPage>e51623</RefPage> <RefTotal>Tudur Smith C, Stocken DD, Dunn J, Cox T, Ghaneh P, Cunningham D, Neoptolemos JP. The value of source data verification in a cancer clinical trial. PLoS ONE. 2012;7(12):e51623. DOI: 10.1371&#47;journal.pone.0051623</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1371&#47;journal.pone.0051623</RefLink> </Reference> <Reference refNo="50"> <RefAuthor>Venet D</RefAuthor> <RefAuthor>Doffagne E</RefAuthor> <RefAuthor>Burzykowski T</RefAuthor> <RefAuthor>Beckers F</RefAuthor> <RefAuthor>Tellier Y</RefAuthor> <RefAuthor>Genevois-Marlin E</RefAuthor> <RefAuthor>Becker U</RefAuthor> <RefAuthor>Bee V</RefAuthor> <RefAuthor>Wilson V</RefAuthor> <RefAuthor>Legrand C</RefAuthor> <RefAuthor>Buyse M</RefAuthor> <RefTitle>A statistical approach to central monitoring of data quality in clinical trials</RefTitle> <RefYear>2012</RefYear> <RefJournal>Clin Trials</RefJournal> <RefPage>705-13</RefPage> <RefTotal>Venet D, Doffagne E, Burzykowski T, Beckers F, Tellier Y, Genevois-Marlin E, Becker U, Bee V, Wilson V, Legrand C, Buyse M. A statistical approach to central monitoring of data quality in clinical trials. Clin Trials. 2012 Dec;9(6):705-13. DOI: 10.1177&#47;1740774512447898</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1177&#47;1740774512447898</RefLink> </Reference> <Reference refNo="51"> <RefAuthor>Verhulst K</RefAuthor> <RefAuthor>Artiles-Carloni L</RefAuthor> <RefAuthor>Beck M</RefAuthor> <RefAuthor>Clarke JT</RefAuthor> <RefAuthor>Neto JC</RefAuthor> <RefAuthor>Cox GF</RefAuthor> <RefAuthor>Fernhoff PM</RefAuthor> <RefAuthor>Guffon N</RefAuthor> <RefAuthor>Kong Y</RefAuthor> <RefAuthor>Martins AM</RefAuthor> <RefAuthor>Tylki-Szymanska A</RefAuthor> <RefAuthor>Whitley CB</RefAuthor> <RefAuthor>Wijburg FA</RefAuthor> <RefAuthor>Wraith EJ</RefAuthor> <RefAuthor>Koepper CM</RefAuthor> <RefTitle>Source document verification in the Mucopolysaccharidosis Type I Registry</RefTitle> <RefYear>2012</RefYear> <RefJournal>Pharmacoepidemiol Drug Saf</RefJournal> <RefPage>749-752</RefPage> <RefTotal>Verhulst K, Artiles-Carloni L, Beck M, Clarke JT, Neto JC, Cox GF, Fernhoff PM, Guffon N, Kong Y, Martins AM, Tylki-Szymanska A, Whitley CB, Wijburg FA, Wraith EJ, Koepper CM. Source document verification in the Mucopolysaccharidosis Type I Registry. Pharmacoepidemiol Drug Saf. 2012 Jul;21(7):749-752. DOI: 10.1002&#47;pds.2200</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1002&#47;pds.2200</RefLink> </Reference> <Reference refNo="52"> <RefAuthor>Wu Y</RefAuthor> <RefAuthor>Takkenberg JJ</RefAuthor> <RefAuthor>Grunkemeier GL</RefAuthor> <RefTitle>Measuring follow-up completeness</RefTitle> <RefYear>2008</RefYear> <RefJournal>Ann Thorac Surg</RefJournal> <RefPage>1155-7</RefPage> <RefTotal>Wu Y, Takkenberg JJ, Grunkemeier GL. Measuring follow-up completeness. Ann Thorac Surg. 2008 Apr;85(4):1155-7. DOI: 10.1016&#47;j.athoracsur.2007.12.012</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.athoracsur.2007.12.012</RefLink> </Reference> <Reference refNo="53"> <RefAuthor>Xian Y</RefAuthor> <RefAuthor>Fonarow GC</RefAuthor> <RefAuthor>Reeves MJ</RefAuthor> <RefAuthor>Webb LE</RefAuthor> <RefAuthor>Blevins J</RefAuthor> <RefAuthor>Demyanenko VS</RefAuthor> <RefAuthor>Zhao X</RefAuthor> <RefAuthor>Olson DM</RefAuthor> <RefAuthor>Hernandez AF</RefAuthor> <RefAuthor>Peterson ED</RefAuthor> <RefAuthor>Schwamm LH</RefAuthor> <RefAuthor>Smith EE</RefAuthor> <RefTitle>Data quality in the American Heart Association Get With The Guidelines-Stroke (GWTG-Stroke): results from a national data validation audit</RefTitle> <RefYear>2012</RefYear> <RefJournal>Am Heart J</RefJournal> <RefPage>392-8, 398</RefPage> <RefTotal>Xian Y, Fonarow GC, Reeves MJ, Webb LE, Blevins J, Demyanenko VS, Zhao X, Olson DM, Hernandez AF, Peterson ED, Schwamm LH, Smith EE. Data quality in the American Heart Association Get With The Guidelines-Stroke (GWTG-Stroke): results from a national data validation audit. Am Heart J. 2012 Mar;163(3):392-8, 398.e1. DOI: 10.1016&#47;j.ahj.2011.12.012</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.ahj.2011.12.012</RefLink> </Reference> <Reference refNo="54"> <RefAuthor>Brennan PF</RefAuthor> <RefAuthor>Stead WW</RefAuthor> <RefTitle>Assessing data quality: from concordance, through correctness and completeness, to valid manipulatable representations</RefTitle> <RefYear>2000</RefYear> <RefJournal>J Am Med Inform Assoc</RefJournal> <RefPage>106-7</RefPage> <RefTotal>Brennan PF, Stead WW. Assessing data quality: from concordance, through correctness and completeness, to valid manipulatable representations. J Am Med Inform Assoc. 2000 Jan-Feb;7(1):106-7. DOI: 10.1136&#47;jamia.2000.0070106</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1136&#47;jamia.2000.0070106</RefLink> </Reference> <Reference refNo="55"> <RefAuthor>Dugas M</RefAuthor> <RefAuthor>Lange M</RefAuthor> <RefAuthor>M&#252;ller-Tidow C</RefAuthor> <RefAuthor>Kirchhof P</RefAuthor> <RefAuthor>Prokosch HU</RefAuthor> <RefTitle>Routine data from hospital information systems can support patient recruitment for clinical studies</RefTitle> <RefYear>2010</RefYear> <RefJournal>Clin Trials</RefJournal> <RefPage>183-9</RefPage> <RefTotal>Dugas M, Lange M, M&#252;ller-Tidow C, Kirchhof P, Prokosch HU. Routine data from hospital information systems can support patient recruitment for clinical studies. Clin Trials. 2010 Apr;7(2):183-9. DOI: 10.1177&#47;1740774510363013</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1177&#47;1740774510363013</RefLink> </Reference> <Reference refNo="56"> <RefAuthor>Sorensen HT</RefAuthor> <RefAuthor>Sabroe S</RefAuthor> <RefAuthor>Olsen J</RefAuthor> <RefTitle>A framework for evaluation of secondary data sources for epidemiological research</RefTitle> <RefYear>1996</RefYear> <RefJournal>Int J Epidemiol</RefJournal> <RefPage>435-42</RefPage> <RefTotal>Sorensen HT, Sabroe S, Olsen J. A framework for evaluation of secondary data sources for epidemiological research. Int J Epidemiol. 1996 Apr;25(2):435-42. DOI: 10.1093&#47;ije&#47;25.2.435</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1093&#47;ije&#47;25.2.435</RefLink> </Reference> <Reference refNo="57"> <RefAuthor>Winter A</RefAuthor> <RefAuthor>Funkat G</RefAuthor> <RefAuthor>Haeber A</RefAuthor> <RefAuthor>Mauz-Koerholz C</RefAuthor> <RefAuthor>Pommerening K</RefAuthor> <RefAuthor>Smers S</RefAuthor> <RefAuthor>Stausberg J</RefAuthor> <RefTitle>Integrated information systems for translational medicine</RefTitle> <RefYear>2007</RefYear> <RefJournal>Methods Inf Med</RefJournal> <RefPage>601-7</RefPage> <RefTotal>Winter A, Funkat G, Haeber A, Mauz-Koerholz C, Pommerening K, Smers S, Stausberg J. Integrated information systems for translational medicine. Methods Inf Med. 2007;46(5):601-7. DOI: 10.1160&#47;ME9063</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1160&#47;ME9063</RefLink> </Reference> <Reference refNo="58"> <RefAuthor>Johnson SG</RefAuthor> <RefAuthor>Speedie S</RefAuthor> <RefAuthor>Simon G</RefAuthor> <RefAuthor>Kumar V</RefAuthor> <RefAuthor>Westra BL</RefAuthor> <RefTitle>Application of An Ontology for Characterizing Data Quality For a Secondary Use of EHR Data</RefTitle> <RefYear>2016</RefYear> <RefJournal>Appl Clin Inform</RefJournal> <RefPage>69-88</RefPage> <RefTotal>Johnson SG, Speedie S, Simon G, Kumar V, Westra BL. Application of An Ontology for Characterizing Data Quality For a Secondary Use of EHR Data. Appl Clin Inform. 2016;7(1):69-88. DOI: 10.4338&#47;ACI-2015-08-RA-0107</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.4338&#47;ACI-2015-08-RA-0107</RefLink> </Reference> <Reference refNo="59"> <RefAuthor>Ngouongo SM</RefAuthor> <RefAuthor>L&#246;be M</RefAuthor> <RefAuthor>Stausberg J</RefAuthor> <RefTitle>The ISO&#47;IEC 11179 norm for metadata registries: does it cover healthcare standards in empirical research&#63;</RefTitle> <RefYear>2013</RefYear> <RefJournal>J Biomed Inform</RefJournal> <RefPage>318-27</RefPage> <RefTotal>Ngouongo SM, L&#246;be M, Stausberg J. The ISO&#47;IEC 11179 norm for metadata registries: does it cover healthcare standards in empirical research&#63; J Biomed Inform. 2013 Apr;46(2):318-27. DOI: 10.1016&#47;j.jbi.2012.11.008</RefTotal> <RefLink>http:&#47;&#47;dx.doi.org&#47;10.1016&#47;j.jbi.2012.11.008</RefLink> </Reference> </References> <Media> <Tables> <Table format="png"> <MediaNo>1</MediaNo> <MediaID>1</MediaID> <Caption><Pgraph><Mark1>Table 1: List of indicators proposed by the guideline</Mark1></Pgraph></Caption> </Table> <Table format="png"> <MediaNo>2</MediaNo> <MediaID>2</MediaID> <Caption><Pgraph><Mark1>Table 2: List of indicators identified in the literature review</Mark1></Pgraph></Caption> </Table> <NoOfTables>2</NoOfTables> </Tables> <Figures> <NoOfPictures>0</NoOfPictures> </Figures> <InlineFigures> <NoOfPictures>0</NoOfPictures> </InlineFigures> <Attachments> <Attachment> <MediaNo>1</MediaNo> <MediaID filename="mibe000199.a1.pdf" mimeType="application/pdf" origFilename="mibe000199&#95;Attachment1.pdf" size="72198" url="">1</MediaID> <AttachmentTitle>Definition of indicator completeness</AttachmentTitle> </Attachment> <NoOfAttachments>1</NoOfAttachments> </Attachments> </Media> </OrigData> </GmsArticle>