mibe000215
10.3205/mibe000215
urn:nbn:de:0183-mibe0002156
Research Article
Comparative evaluation of automated information extraction from pathology reports in three German cancer registries
Vergleichende Analyse automatisierter Informationsextraktion aus Pathologieberichten in drei deutschen Krebsregistern
Schulz
Schulz
Stefan
S
Averbis GmbH, Salzstraße 15, 79098 Freiburg im Breisgau, GermanyAverbis GmbH, Freiburg, Germany
stefan.schulz@averbis.com
author
Fix
Fix
Sonja
S
Averbis GmbH, Freiburg, Germany
author
Klügl
Klügl
Peter
P
Averbis GmbH, Freiburg, Germany
author
Bachmayer
Bachmayer
Tamira
T
Klinische Landesregisterstelle des Krebsregisters Baden-Württemberg, Stuttgart, Germany
author
Hartz
Hartz
Tobias
T
Klinisches Krebsregister Niedersachsen, Hannover, Germany
author
Richter
Richter
Martin
M
Klinische Landesregisterstelle des Krebsregisters Baden-Württemberg, Stuttgart, Germany
author
Herm-Stapelberg
Herm-Stapelberg
Nils
N
Krebsregister Rheinland-Pfalz, Mainz, Germany
author
Daumke
Daumke
Philipp
P
Averbis GmbH, Freiburg, Germany
author
German Medical Science GMS Publishing House
Düsseldorf
610
cancer registries
text mining
Krebsregister
Text mining
20210331
engl
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License.
Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung).
1860-9171
17
1
GMS Medizinische Informatik, Biometrie und Epidemiologie
GMS Med Inform Biom Epidemiol
01
German Ministry for Education and Research (BMBF)
German Ministry for Education and Research (BMBF)
Das Anreichern von Krebsregistern mit Daten aus medizinischen Texten, bei gleichzeitiger Sicherstellung der Datenqualität, ist aufgrund der Heterogenität dieser Quellen mit erheblichem Aufwand verbunden. Die Unterstützung dieser Aufgabe durch IT-Lösungen soll diesen Prozess beschleunigen und optimieren. Zu diesem Zweck wurde das kommerzielle Text-Mining-System Averbis Health Discovery darauf zugeschnitten, Freitextinformationen für das Krebsregister Baden-Württemberg zu verwerten. Die folgenden Entitätstypen wurden aus Befundberichten der Pathologie extrahiert: Tumorlokalisation und -morphologie, pTNM, Grading, untersuchte und betroffene (Sentinel-) Lymphknoten, Lateralität und R-Klassifikation. Je nach Tumortyp wurden verschiedene Ansätze des maschinellen Lernens sowie Regeln für die Tumorentitäten Brust, Prostata, Kolon/Rektum und Haut verwendet. Während für den Pilotstandort die F-Werte zwischen 0,804 und 0,996 lagen, fielen die Werte, wenn die Extraktionspipeline auf zwei neue Standorte (Krebsregister Rheinland-Pfalz und Niedersachsen) angewendet wurde, für die Morphologie von 0,950 auf 0,657 und 0,933 und für die Lokalisierung (Topographie) von 0,902 auf 0,675 und 0,768. Es gab viel weniger Unterschiede in der R-Klassifikation und bei den Lymphknoten. Eine gründliche Fehleranalyse verwies auf zahlreiche Probleme, die diese Unterschiede erklären, z.B. unterschiedliche Arbeitsabläufe zwischen den Standorten, Widersprüche zwischen Text und codiertem Inhalt sowie unterschiedlicher Umgang mit fehlenden Werten.
Feeding cancer registries with data extracted from textual reports, while maintaining a high level of data quality, has always been a labour-intensive task, due to the heterogeneity of the sources. The support of this task by IT solutions is expected to accelerate and optimise this process. To this end, the commercial text mining system Averbis Health Discovery was tailored to extract information from free text at the cancer registry of the federal state of Baden-Württemberg. The following entity types were extracted from German-language pathology reports: tumour localisation and morphology, pTNM, grading, (sentinel) nodes examined and affected, laterality and R-class. According to the entity type, several machine learning approaches as well as rules were used for the tumour types breast, prostate, colorectal and skin. Whereas for the pilot site, F values ranged between 0.800 and 0.996, values dropped when applying the extraction pipeline to two new sites (cancer registries Rhineland-Palatinate and Lower Saxony), for morphology from 0.950 to 0.657 and 0.933, and for localisation (topography) from 0.902 to 0.675 and 0.768. There was much less difference with R-class and lymph node counts. A thorough error analysis revealed numerous issues that explain these differences, such as different workflows between the sites, disagreements between textual and coded content as well as different handlings of missing values.
1 Introduction1.1 Medical registriesMedical registries are databases fed by uniform data about a particular health condition. Their purposes encompass monitoring and improvement of quality of care, clinical and epidemiological research as well as questions related to health economics. Cancer registries constitute the most important type of registries. According to the European Network of Cancer Registries (ENCR), there are two major aims of population-based cancer registries, viz. (i) the collection of data on new cancer cases in a defined geographic region with the purpose to investigate the burden of specific malignant disorders, and (ii) to provide a basis for investigating cancer aetiology and outcome (incidence, prevalence or survival). Thus, cancer registries are important means to assess the impact and effectiveness of health interventions, both regarding the provision of diagnostic and therapeutic measures, and the implementation of preventive actions by public policies. 1.2 Cancer registries in GermanyIn Germany, cancer registries support treatment and follow-up care . They are divided into clinical cancer registries, which describe hospital populations, and population-based (aka epidemiological) ones, which track cancer cases across institutions. Cancer registries are organised at the level of the 16 German federal states. The cases underlying this study cover the federal states of Baden-Württemberg (BW), Rhineland-Palatinate (RP) and Lower Saxony (LS). Table 1 provides some basic information about these three institutions and the population they support.1.2.1 DatasetsThe three registries receive notifications by hospitals, clinics and doctor’s offices. They exchange data with civil registration offices in order to facilitate record linkage and notifications of deaths. Their dependence on manual annotation of cancer notifications requires detailed workflows and annotation guidelines. Input data are mainly narratives, with varying degrees of embedded codes (ICD-10, ICD-O, TNM). Some cancer centres at university hospitals produce well-curated structured datasets for their internal quality assurance processes, which, as a side effect, constitute a valuable input for the state-level registries. Target datasets vary between tumour types but there is a common set of fields and value restrictions that are equally valid for the majority of tumour types: ICD-10 – in its German modification ICD-10-GM – is used for disease coding, enhanced by ICD-O for morphology and localisation (also known as topography) of primary tumours and metastases. The TNM system is used for primary tumour size (T), regional lymph nodes (N) and distant metastases; tumour grading is expressed by a score between G1 (low grade) and G4 (very high grade), including Gx (undetermined grade). Datasets also include lymph node count values, distinguishing between examined and affected ones, for all lymph nodes and for sentinel lymph nodes in particular. For all tumour localisations, laterality values are given (left, right, bilateral, midline, not applicable). Finally, the UICC R classification denotes absence or presence of residual tumours after treatment. 1.2.2 WorkflowsIn the three centres, two parallel workflows can be distinguished. The first workflow starts with the submission of structured and appropriately formatted hospital data to the registry’s data management team, whose work is therefore limited to minor data curation. In contrast, the second workflow requires manual annotations. It is this workflow where our effort is centred and for which results are described in this paper. It starts with the reception of narrative pathology reports (see Table 1 ), which have to be read by experts, who then assign codes and values, which requires time and considerable intellectual efforts. As long as both sources describe the same patients, sources are merged and “best of” registry entries are created, according to the interpretation by the registry experts. Evaluations have shown that quality improves significantly when pathology report annotations are added to the data from the hospitals , . However, there are still a number of bottlenecks, particularly regarding quality issues at the source. Legal notification requirements in Germany are recent (2009 for hospitals, 2011 for doctor’s offices and labs, e.g. pathology), and therefore software that is in current use still lacks maturity, with some modules in beta status. Process quality at the registries is also affected by a lack of qualified staff and by a constantly changing environment with frequent updates of standard operating procedures and coding guidelines. The creation of “best of” adjudications is complex and still error-prone, due to the lack of standardisation of textual input.1.2.3 ChallengesWhenever structured data is manually entered into forms, after reading and analysing free text reports, quality issues are unavoidable and need to be monitored. Despite considerable efforts regarding quality, such measures have not yet been implemented in routine by the three cancer registries. Ideally, double annotations would allow the monitoring of inter-annotator agreement (e.g. by Kappa statistics) and subsequent adjudication of diverging annotations. This would allow ongoing quality assessment at annotator level, which then triggers training and coaching activities, as well as the ongoing refinement of annotation guidelines. Inter-annotator agreement values are also important reference data for assessing the outcome of machine annotations. If language models are trained by inconsistently annotated data, performance measures achieved by information extraction performed by a text mining system cannot be expected higher than the threshold of human agreement. Due to resource constraints in the cancer registries, this study did not allow to measure interrater agreement, which can be seen as a limitation for the evaluation of the text mining tools we describe in the following sections. However, existing inter-annotator agreement studies for cancer registry coding in a German context have resulted in 76.7% for ICD-O tumour localisation, 80.3% for ICD-O morphology, 98.5% for ICD-O behaviour and 73.3% for ICD-10, measured by Fleiss’ Kappa and considering full agreement across all hierarchical levels . 1.3 Purpose of the studyGiven the high workload required by the manual analysis of pathology texts, the purpose of this study is to assess how human language technologies, powered by state-of-the-art artificial intelligence methodologies can support this task. Such support can be capitalised by the following scenarios:fully automated filling of registries by identifying relevant text passages in pathology reports,support of manual data input by tentative pre-filling of fields, automated clustering by tumour type (e.g. breast, lung, colorectal, prostate),keyword highlighting to accelerate manual annotation, content-based filtering, e.g. filtering by T4-tumours.In this paper we will assess the quality of machine-based analysis of pathology texts against a gold standard, i.e. to which extent it fits the requirements of the registries. The question which of the above scenarios can be supported would then be subject of a future implementation study.
2 Background2.1 Human language technologies applied to medical textsNatural language processing (NLP), particularly information extraction , together with terminology and ontology , have been awarded increasing attention in healthcare and biomedical research due to the predominance of free text in electronic health records, opposed to the need for structured and standardised data, e.g. for observational research, health statistics, disease reporting, quality assurance and billing. Whereas the field of biomedical terminology systems is huge, tumour documentation has internationally converged to a relatively small set of terminological (quasi-)standards, with <![CDATA[ICD-10</PlainText></TextGroup> for macroscopic aspects, ICD-O for morphology and localisation, and TNM (version 7/8) for staging. Depending on the tumour type, additional scoring systems and clinical stage classifications are added, such as the Gleason score for prostate tumours, or the Clark and Breslow staging systems for skin tumours.</Pgraph><Pgraph>NLP methods that automatically assign such codes have shown considerable progress, which, however, critically depends on available resources like annotated corpora and language-specific vocabularies. Apart from English, all natural languages suffer from a lack of these resources, which explains the underuse of NLP methods despite increasing demand. For German, important medical terminology resources lack localisation (e.g. SNOMED CT <TextLink reference="8"></TextLink> and the NCI thesaurus <TextLink reference="9"></TextLink>). Another important reason for the underuse of NLP on clinical content is the extreme brevity of clinical texts, characterised by acronyms and abbreviations, which require disambiguation efforts, unless under a very limited scope with a specific vocabulary.</Pgraph><Pgraph>Due to data privacy and therefore the lack of public availability of clinical texts – in contradistinction to scientific text corpora like the millions of MEDLINE abstracts and PubMedCentral full texts, clinical language processing has mostly occurred inside closed information environments, with amounts of data orders of magnitude below the size of openly available corpora. This constitutes a limit to the use of recent approaches from the field of machine learning – particularly deep learning <TextLink reference="5"></TextLink>, which are currently revolutionising the way human language is processed by computers. Wherever sufficiently large (annotated) amounts of data are available to train specific neural networks (NNs), models trained accordingly have outperformed traditional machine learning methods. Nevertheless, there is also evidence that manually-crafted rules for specific extraction tasks can achieve better results <TextLink reference="10"></TextLink>. These findings should be taken into account when processing clinical texts. In addition, manual annotation of clinical texts is labour-intensive, so that the limits of what is possible are quickly reached with supervised learning methods (i.e. those depending on human annotation), given the amount of annotations necessary to achieve the desired outcome. </Pgraph></TextBlock>
<TextBlock linked="yes" name="3 Material and methods">
<MainHeadline>3 Material and methods</MainHeadline><SubHeadline>3.1 Textual data</SubHeadline><Pgraph>Our analysis is focused on pathology reports for four types of cancer (breast, prostate, colon/rectum, skin). These documents are usually sent to the cancer registries by pathology labs as PDF files and do not follow any standardised template. All documents are in German. Several documents may refer to the same patient, but do not necessarily arrive in the order they have been created. </Pgraph><Pgraph>An important aspect is also that the source documents used in this study are not representative for all pathology reports produced in each state. This is due to the data curation activities done by university hospitals (workflo<TextGroup><PlainText>w 1</PlainText></TextGroup> in 1.2.2). As a consequence, their source documents do not reach the cancer registries, which receive structured data instead. </Pgraph><Pgraph>Table 2 <ImgLink imgNo="2" imgType="table"/> shows the free-text document sets that were provided by the cancer registries. One document corresponds to one pathology report.</Pgraph><SubHeadline>3.2 Structured data</SubHeadline><Pgraph>All structured data used for this study were strictly related to datasets corresponding to exactly one document. According to the workflow, the dataset belongs either to a tumour diagnosis record or a tumour progress record. The fields of the datasets as given in Table 3 <ImgLink imgNo="3" imgType="table"/> were used for the comparison with the automatic text analysis in our study. </Pgraph><Pgraph>The following applies to cancer registry data sets: d<Subscript>all</Subscript> documents and structured r<Subscript>all</Subscript> records ( |d<Subscript>all</Subscript>| = |r<Subscript>all</Subscript>|). In order to guarantee representativeness, the selection is done according to the following criteria:</Pgraph><Pgraph><UnorderedList><ListItem level="1">coverage of a predefined period, </ListItem><ListItem level="1">cancer types: breast, prostate, colon/rectum, skin,</ListItem><ListItem level="1">all documents annotated in this period (or a random sample thereof),</ListItem><ListItem level="1">only documents that were annotated or are scheduled for being annotated.</ListItem></UnorderedList></Pgraph><Pgraph>The following data were excluded for being considered out of scope for this study:</Pgraph><Pgraph><UnorderedList><ListItem level="1">structured data that were not extracted from a document by the cancer registries (including structured data that arrive at the registries together with a document, but where no data extraction needs to be done at the registries),</ListItem><ListItem level="1">structured data for which no document was available,</ListItem><ListItem level="1">data related to any cancer type other than breast, prostate, colorectal or skin,</ListItem><ListItem level="1">the number of documents per patient is not taken into account, because the study is on documents, not on patients. </ListItem></UnorderedList></Pgraph><Pgraph>The rationale for this study design is that we exclusively scrutinise the assigning of predefined codes or values to narrative pathology reports. The hypothesis to be tested is whether an NLP-based text mining approach extracts information items in a quality comparable to human experts.</Pgraph><SubHeadline2>3.2.1 Development and test set</SubHeadline2><Pgraph>The textual and structural data provided by the cancer registry of Baden-Württemberg (BW) is utilized for the initial customization of a commercial text mining software. The overall dataset is divided into a development set with 44,324 reports and a test set with 2,057 reports. The development set is used for the engineering, adaptation and optimization of the rule-based components as well as for the training of the machine learning models. The BW test set is only applied for evaluation.</Pgraph><Pgraph>The remaining two datasets of RP (Rhineland-Palatinate state, 26,894 reports) and LS (Lower Saxony state, 19,170 reports) are only utilized for evaluating the existing text mining system and thus provide a suitable experimental setting to investigate its generalizability.</Pgraph><SubHeadline>3.3 Tools </SubHeadline><Pgraph>The text mining technology used is AHD (Averbis Health Discovery) <TextLink reference="11"></TextLink> by Averbis GmbH <TextLink reference="12"></TextLink>. It contains over 50 different text mining annotators, e.g. for the recognition of diagnostic statements, medical procedures, lab values, drugs, anatomy, morphology, scores and others. Available for several languages, including English and German, AHD bundles annotators in predefined text mining pipelines tailored to document types like discharge summaries or pathology reports. AHD has been successfully used for various use cases, e.g. for data driven patient recruitment for clinical trials <TextLink reference="13"></TextLink>, automated coding and billing <TextLink reference="14"></TextLink>, documentation support in private practices <TextLink reference="15"></TextLink>, rare disease identification <TextLink reference="16"></TextLink>, antibiotic resistance monitoring <TextLink reference="17"></TextLink>, radiology report analysis <TextLink reference="18"></TextLink>, and health data de-identification <TextLink reference="19"></TextLink>. Health Discovery is based on the Unstructured Information Management Architecture (UIMA) <TextLink reference="20"></TextLink>, an extensible framework for text analysis and text processing. It stresses the interoperability of components (analysis engines), which communicate in a pipeline by adding or modifying the meta information stored in the Common Analysis System (CAS), which contains the currently processed document. This information is represented by typed feature structures and indexed for efficient access. The most common type of a feature structure is the annotation, which assigns its type and additional features to a span of text. Most analysis engines create new annotations or modify existing ones in order to represent the result of their analysis.</Pgraph><SubHeadline>3.4 Experimental approach</SubHeadline><SubHeadline2>3.4.1 Pathology pipeline</SubHeadline2><Pgraph>AHD’s pathology pipeline is designed for extracting information from pathology reports and addresses challenges typical for this text genre, which often contains sensitive patient data or other protected health information. This explains why there are no public datasets for applying machine-learning-based approaches. Information needed for pathology coding may not be explicitly present in the document, but needs to be derived from different, interrelated information. Finally, coding rules may differ from cancer type to another. </Pgraph><Pgraph>To address these challenges, the pathology pipeline combines different approaches and sources to deliver high-quality coding results such as </Pgraph><Pgraph><UnorderedList><ListItem level="1">terminology concept mapping (ICD-10, ICD-O…) and wordlists,</ListItem><ListItem level="1">rule-based annotations,</ListItem><ListItem level="1">machine learning,</ListItem><ListItem level="1">algorithms for combining the approaches.</ListItem></UnorderedList></Pgraph><Pgraph>The AHD pathology pipeline is composed of several general purpose text mining components (e.g. for disambiguation) as well as specialized ones for clinical documents. On top of this, a specialized component addresses cancer-specific coding on document level. </Pgraph><Pgraph>In this section, we first describe the structure of the pathology pipeline concerning the more general components and then the component for cancer coding in more detail. Table 4 <ImgLink imgNo="4" imgType="table"/> describes the pipeline, with its components, which are sequentially processed. Figure 1 <ImgLink imgNo="1" imgType="figure"/> shows a screenshot of the interface of the pathology pipeline within the AHD text mining system.</Pgraph><Pgraph>The pipeline components implement methodologies ranging from rules and dictionaries to different kinds of machine learning approaches. Terminology-based annotators for tumour morphology and localisation apply dictionary lookup with linguistic pre-processing, abbreviation resolving and other extensions. Machine learning includes shallow approaches like Maximum entropy models, Conditional Random Fields and Support Vector Machines on the one hand and deep learning approaches like Convolutional Neural Networks on the other hand. Many components rely on manually engineered rules, which are still the method of choice where training data are scarce or annotations too expensive. The rules are created using UIMA Ruta <TextLink reference="21"></TextLink>, which provides a mixture of declarative and imperative paradigms to specify patterns of arbitrary annotations and their consequences. Finally, some components simply are composed of specific algorithms implemented in a programming language.</Pgraph><SubHeadline2>3.4.2 Pathology document classification</SubHeadline2><Pgraph>Our document classification component for cancer registry documents assigns the 13 output fields (see Table 3 <ImgLink imgNo="3" imgType="table"/>) and is composed of multiple subcomponents. For an initial categorization, support vector machines (SVMs) classify the pathology report into one of the supported cancer types, i.e. breast, prostate, skin, rectum/colon. This information feeds downstream processing steps for cancer-specific fall-back values or coding rules.</Pgraph><Pgraph>Morphology and tumour localisation are then separately classified using convolutional neural networks (CNNs), independently of the cancer type identified. If no valid output is identified above a given threshold, different fall-back procedures are applied for both tumour morphology and localisation: First, concepts for either type that have been annotated by dictionary lookup are ranked using SVMs, and the code with the highest confidence in its context is selected for further processing. If no concepts have been extracted, a fall-back value depending on the cancer type is chosen. The final ICD-O codes are then assigned given a predefined mapping to ICD-10-GM diagnosis codes. </Pgraph><Pgraph>The remaining entity types like tumour, node, metastasis, grading, laterality and lymph nodes are selected using only the pre-processing functionality of the AHD pipeline. Specialized annotators detect potentially all related text mentions. For multiple mentions, either the highest value (e.g., grading) is selected or a majority voting is applied (e.g., laterality).</Pgraph><Pgraph>Finally, a set of additional rules perform a final modification of the values by using all given information. This post processing adapts the coding to the cancer registry coding guideline and induces values not directly represented in the pathology report. One example is the rule that prostate grading is not provided by TNM but by the Gleason score, or that laterality only matters for breast cancer (the other tumour sites are not bilateral), but that for both prostate and colorectal cancer the guideline requires the laterality value set to “T”. Figure 2 <ImgLink imgNo="2" imgType="figure"/> depicts the workflow.</Pgraph><Pgraph>For machine learning models, SVMs and CNNs are used, supported by the following formative evaluations on the BW dataset. The first ones, in particular multiple one-vs-all linear SVMs are applied for the initial categorization of the cancer type by single-label multi-class document classification. The features only consist of bag-of-stems and are weighted using logarithmic frequencies with redundancy <TextLink reference="22"></TextLink> with L2-normalization. The classification model for the pipeline was trained using a subset of the BW training set (37,880 reports). A separate five-fold cross evaluation on this collection highlights the applicability of the model for the given task. The results in <TextGroup><PlainText>Table 5</PlainText></TextGroup> <ImgLink imgNo="5" imgType="table"/> indicate that classical (“shallow”) machine learning is sufficient for this task. </Pgraph><Pgraph>A “deep” neural ML approach, viz. CNNs are applied to the classification of tumour morphology and localisation via single-label multi-class document classification. The network architecture is based on Kim et al. <TextLink reference="23"></TextLink> using TensorFlow <TextLink reference="24"></TextLink>. As an input space, fastText word embeddings with subword information provide a good trade-off between classification accuracy and prediction speed. The embeddings are tuned on the given data and classification task during the training of the model. The hyperparameters of the CNNs are optimized using BOHB [30], a combination of Bayesian Optimization and Hyperband. The models are trained on the complete train set of BW (d<Subscript>train</Subscript>). Evaluation results (F1 score) on the test set comparing the performance to a baseline support vector machine (SVM) are depicted in Table 6 <ImgLink imgNo="6" imgType="table"/>. This clearly supports our option for using CNNs for this task.</Pgraph></TextBlock>
<TextBlock linked="yes" name="4 Results">
<MainHeadline>4 Results</MainHeadline><Pgraph>Text mining quality was evaluated for the features of <TextGroup><PlainText>Table 4 </PlainText></TextGroup><ImgLink imgNo="4" imgType="table"/> by comparing the automatic recognition with the manual coding results of the cancer registries. For each feature, the F1 score (a harmonic mean of precision and recall) was calculated. It was assumed that the manual coding results were correct and could therefore be considered the gold standard. However, we will see that this assumption is relativized by a detailed error analysis, as an important aspect of the results we report.</Pgraph><SubHeadline>4.1 Overview</SubHeadline><Pgraph>Figure 3 <ImgLink imgNo="3" imgType="figure"/> shows the evaluation results of the three cancer registries across all features. For 9 of 13 features, the F1 scores are highest for the BW state. </Pgraph><Pgraph>In the following, the results for each of the three cancer registries (BW, RP, LS) are explained and interpreted in detail.</Pgraph><SubHeadline>4.2 Evaluation Baden-Württemberg state (BW)</SubHeadline><SubHeadline2>4.2.1 Results</SubHeadline2><Pgraph>The evaluation of the BW cancer registry was performed on the test dataset n=2,057. The results show an F1 score greater than 0.8 for all features, using the manual coding results as ground truth (Table 7 <ImgLink imgNo="7" imgType="table"/>). The lymph node status features are distinguished by particularly high F1 score values with a match of up to 99.6%. The lowest value is found for the R-Class feature. </Pgraph><Pgraph></Pgraph><SubHeadline2>4.2.2 Gap analysis</SubHeadline2><Pgraph>For assessing gaps in the pathology feature extraction, the top 10 mismatches of the above mentioned test data set (n=2,057) were reviewed by experts at the feature level (Table 8 <ImgLink imgNo="8" imgType="table"/>). Since “diagnosis” is derived from tumour morphology and localisation, it is not included in this analysis. </Pgraph><Pgraph>The deviations can be summarized in six categories (<TextGroup><PlainText>Table 9 </PlainText></TextGroup><ImgLink imgNo="9" imgType="table"/>). </Pgraph><SubHeadline>4.3 Evaluation Rhineland-Palatinate state (RP)</SubHeadline><SubHeadline2>4.3.1 Results</SubHeadline2><Pgraph>The evaluation of the RP dataset shows F1 score values between 0.6153 (Grading) and 0.9725 (Sentinel Nodes affected), cf. Table 10 <ImgLink imgNo="10" imgType="table"/>. In particular, the scores for tumour morphology and localisation, as well as ICD-10 diagnosis are up to 0.29 lower when compared with the BW results. </Pgraph><SubHeadline2>4.3.2 Gap analysis</SubHeadline2><Pgraph>For a variance analysis in the pathology feature extraction, the top 10 mismatches of the above-mentioned data set (n=26,894) were reviewed by experts at the feature level (Table 11 <ImgLink imgNo="11" imgType="table"/>). Where appropriate, sources of error were aggregated. Again, “diagnosis” is not considered in the gap analysis as this feature is derived from the morphology and topography codes.</Pgraph><Pgraph>The deviations can be summarized in eight categories (Table 12 <ImgLink imgNo="12" imgType="table"/>).</Pgraph><SubHeadline>4.4 Evaluation Lower Saxony state (LS)</SubHeadline><SubHeadline2>4.4.1 Results</SubHeadline2><Pgraph>The evaluation of the LS dataset shows F-values between 0.768 (Nodes tested) and 0.9836 (Sentinel Nodes affected), cf. Table 13 <ImgLink imgNo="13" imgType="table"/>. Similar to RP data, particularly for tumour morphology, localisation and diagnosis, values are significantly lower compared to BW (up to 0.141), although not to the same extent as in the RP dataset. </Pgraph><SubHeadline2>4.4.2 Gap analysis</SubHeadline2><Pgraph>Again, the top 10 mismatches of the above-mentioned data set (n=19,170) were reviewed by experts at the feature level (Table 14 <ImgLink imgNo="14" imgType="table"/>). Where appropriate, sources of error were aggregated. The feature “diagnosis” is not considered in the gap analysis as it is derived from the morphology and topography codes. </Pgraph><Pgraph>The deviations can be summarized in six categories (<TextGroup><PlainText>Table 15 </PlainText></TextGroup><ImgLink imgNo="15" imgType="table"/>).</Pgraph></TextBlock>
<TextBlock linked="yes" name="5 Discussion">
<MainHeadline>5 Discussion</MainHeadline><Pgraph>Figure 3 <ImgLink imgNo="3" imgType="figure"/> shows for most of the features that the BW dataset produced the best result, followed by LS and RP. This is not surprising because BW is the first cancer registry that is already using AHD in their daily business. In several iterations, all models and rules had been tailored to these data. Data also show that the results of the features extracted by rules were more robust compared to those based on models trained on the BW dataset. Tumour localisation and morphology F-values suffered severe drops up to 30%. Only the rule-based grading and laterality values showed a similar behaviour. </Pgraph><Pgraph>The BW error analysis shows that automatic highly accurate text recognition, with deviations from the manual coding results are within the range of interrater reliability. Most deviations do not result from text mining errors, but from the application of coding rules by the coding staff. Some of these rules can easily be reengineered using rule-based NLP methods.</Pgraph><Pgraph>In the LS and RP datasets, many errors could also be traced back to a lack of compliance with coding rules, or slightly different local preferences. In addition, it was evident that the diversity of language in pathology reports prevented that ML procedures performed equally well. Inconsistencies in the gold standards, especially for morphology, reinforce this. In addition, differences in fall-back values (which are assigned in case of missing data) explain disagreements (T, U, X are not always used congruently). Nevertheless, we have to admit that the results for ICD-O morphology and localisation codes are suboptimal. Larger and more representative training corpora will be required, as well as a better implementation of quality mechanisms and rules that manage borderline cases. Just to cite one example: the use of the residual category C50.9 (unspecified location of breast cancer) should raise suspicion, since it is hardly plausible that this important information is missing in the original data. </Pgraph><Pgraph>Grading, node and metastasis status are other frequent sources of gold standard errors. To quantify the influence of these factors and, as a consequence, to assess the overall performance of the AHD pipeline would, however, require inter-coder agreement values for each dataset. Given the absence of such data, the only source of information was the qualitative inspection, which nevertheless provided strong evidence that </Pgraph><Pgraph><OrderedList><ListItem level="1" levelPosition="1" numString="1.">the current gold standard has its shortcomings, so that the information extracted by AHD was often correct whilst the gold standard was wrong, </ListItem><ListItem level="1" levelPosition="2" numString="2.">a limited set of adjustments should significantly improve the text mining performance, and that </ListItem><ListItem level="1" levelPosition="3" numString="3.">difficult boundary issues between tumour localisation and morphology codes, with known disagreements between human coders will always set a limit to the F-values to be expected. </ListItem></OrderedList></Pgraph><Pgraph>In summary, it is noticeable that the ML methods trained on the BW dataset perform better on the LS than on the RP dataset, and for the feature “morphology” even comparably well as for BW. A large part of the deviations is due to missing information in the gold standard or diverging fall-back values. If the evaluation is adjusted for these factors, there are good reasons to assume that the actual recognition quality in all features is already 90% and higher.</Pgraph><Pgraph>The institutional contexts and workflows make it difficult to compare our results to others’. A very recent work <TextLink reference="25"></TextLink> compared multitask learning with single-task learning using CNNs and achieved better results for the multitask approach with up to 60% correctly classified cases according to morphology, localisation, laterality, grade and behaviour, together. Taking ICD-O tumour localisation and morphology, alone, micro-averaged F1 scores amounted to 0.915 and 0.776, respectively, using single-task CNNs. Multitask CNNs (applied to all the five tasks) increased the results by about three percentage points. Multitask CNN results cannot be compared to our study, where CNNs were only used for tumour localisation and morphology. More important, however, is the fact that Alawad et al. <TextLink reference="25"></TextLink> used only three-character codes for tumour localisation and four character codes for morphology. Thus, they used a restricted set of about 130 classes, which eliminated many of the boundary issues we discussed. No inter-coder agreement values were provided. </Pgraph><Pgraph>CNNs also showed the best performance in a study on identification of ICD-O localisation codes for breast and lung cancer. With a small set of 12 different 4-character codes <TextLink reference="26"></TextLink>, with F1 results between 0.722 and 0.811. </Pgraph></TextBlock>
<TextBlock linked="yes" name="6 Conclusion and outlook">
<MainHeadline>6 Conclusion and outlook</MainHeadline><Pgraph>From a technical perspective, the results of this study demonstrate that a hybrid solution that includes machine learning, terminology mapping and rules produces the best results. Whereas challenges such as clinical coding itself are not explicitly addressed, we could demonstrate that the several pieces of information to be extracted had to be interpreted in combination, so that a layered, pipeline-based approach turned out to be the most appropriate.</Pgraph><Pgraph>Technical challenges included different and non-standardised spelling variants and notations, particularly mentions of location and laterality, as well as the correct identification of information items and the way they were related, e.g. locations with tumour types. </Pgraph><Pgraph>The work was done under field conditions, which means that the need for high quality reference datasets – a fundamental requirement for effective training of machine learning models – had to be satisfied by routine data that had grown from practice over long periods, during which annotation rules have changed, many coding staff were involved, and strict quality control measures were only partly put in place. Although no systematic inter-coder agreement data were available from this data, other studies have shown major disagreements between coders for both ICD-O tumour morphology and localisation.</Pgraph><Pgraph>Pathologists usually show little interest in ICD-O coding and structured data collection in general, as long as no financial or scientific incentives are given. This explains conflicting information between textual and semi-structured representations in the same pathology dataset. Fine-grained and coarse-grained information coexists, e.g., a breast quadrant specified in the text along with the “unspecified” C50.9 code, or a precise textual description of a tumour morphology along with the unspecific code “adenocarcinoma or the like”. Trained with this kind of data, even the best possible ML system will underperform and good ML decisions will be classified as wrong. </Pgraph><Pgraph>Language models and rules tailored to one cancer registry will continue underperforming at another cancer registry as long as coding rules and heuristics (e.g. how to deal with missing data) are not the same. Harmonisation at a national level is a desideratum, regardless of the use of manual, automated or hybrid methods. Lack of harmonisation affects data comparability. Nevertheless, we can state that text mining has proven a very suitable approach to support resource-intensive manual coding and to contribute to quality control and coding standardisation. To this end, text mining tools are of high heuristic value, as they bring quality gaps and inconsistent coding rules to the surface. </Pgraph><Pgraph>Along these lines, a next step would then consist in iterative cycles that combine text mining with manual inspection and disagreement analysis. The scope of model training and validation should be broadened by including data from other cancer registries and institutions. Once such a new workflow has proven its usefulness by increasing performance values, text mining routines can be a<TextGroup><PlainText>dapt</PlainText></TextGroup>ed to new functionalities and additional tumour types.</Pgraph></TextBlock>
<TextBlock linked="yes" name="Notes">
<MainHeadline>Notes</MainHeadline><SubHeadline>Funding</SubHeadline><Pgraph>Development, maintenance and customisation of Averbis Health Discovery have been co-funded by the German Ministry for Education and Research (BMBF) through the research networks MIRACUM (01ZZ1801C) and SMITH (01ZZ2003). </Pgraph><SubHeadline>Competing interests</SubHeadline><Pgraph>Philipp Daumke is CEO of Averbis GmbH, the company that develops the tool used for this study. Some of the authors (Sonja Fix, Peter Klügl and Stefan Schulz) are or have been employed by Averbis GmbH.</Pgraph><SubHeadline>Ethics statement</SubHeadline><Pgraph>The ethical foundations for collecting and analysing patient-specific tumour data is covered by the state cancer registry laws of the German federal states that participated in the studies. Averbis’ work was carried out as part of a data processing contract with the three cancer registries. Only de-identified texts were used for the study. </Pgraph></TextBlock>
<References linked="yes">
<Reference refNo="25">
<RefAuthor>Alawad M</RefAuthor>
<RefAuthor>Gao S</RefAuthor>
<RefAuthor>Qiu JX</RefAuthor>
<RefAuthor>Yoon HJ</RefAuthor>
<RefAuthor>Blair Christian J</RefAuthor>
<RefAuthor>Penberthy L</RefAuthor>
<RefAuthor>Mumphrey B</RefAuthor>
<RefAuthor>Wu XC</RefAuthor>
<RefAuthor>Coyle L</RefAuthor>
<RefAuthor>Tourassi G</RefAuthor>
<RefTitle>Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks</RefTitle>
<RefYear>2020</RefYear>
<RefJournal>J Am Med Inform Assoc</RefJournal>
<RefPage>89-98</RefPage>
<RefTotal>Alawad M, Gao S, Qiu JX, Yoon HJ, Blair Christian J, Penberthy L, Mumphrey B, Wu XC, Coyle L, Tourassi G. Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks. J Am Med Inform Assoc. 2020 Jan;27(1):89-98. DOI: 10.1093/jamia/ocz153</RefTotal>
<RefLink>https://doi.org/10.1093/jamia/ocz153</RefLink>
</Reference>
<Reference refNo="1">
<RefAuthor>Altmann U</RefAuthor>
<RefAuthor>Garbe C</RefAuthor>
<RefAuthor>Hofstädter F</RefAuthor>
<RefAuthor>Katalinic A</RefAuthor>
<RefTitle>Standortbestimmung: Klinische und epidemiologische Krebsregister in Deutschland</RefTitle>
<RefYear>2007</RefYear>
<RefJournal>Forum</RefJournal>
<RefPage>28-9</RefPage>
<RefTotal>Altmann U, Garbe C, Hofstädter F, Katalinic A. Standortbestimmung: Klinische und epidemiologische Krebsregister in Deutschland. Forum. 2007;22:28-9.</RefTotal>
</Reference>
<Reference refNo="4">
<RefAuthor>Lappe L</RefAuthor>
<RefAuthor>Ressing M</RefAuthor>
<RefAuthor>Plachky P</RefAuthor>
<RefAuthor>Blettner M</RefAuthor>
<RefAuthor>Zeissig SR</RefAuthor>
<RefTitle>Comparison of coding diagnosis, localisation and histology via ICD-10 and ICD-O-3 between coders and a gold standard</RefTitle>
<RefYear></RefYear>
<RefBookTitle>2018 ENCR Scientific Meeting; 2018 Sep 26-28; Copenhagen, Denmark</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Lappe L, Ressing M, Plachky P, Blettner M, Zeissig SR. Comparison of coding diagnosis, localisation and histology via ICD-10 and ICD-O-3 between coders and a gold standard. In: 2018 ENCR Scientific Meeting; 2018 Sep 26-28; Copenhagen, Denmark.</RefTotal>
</Reference>
<Reference refNo="5">
<RefAuthor>Hahn U</RefAuthor>
<RefAuthor>Oleynik M</RefAuthor>
<RefTitle>Medical Information Extraction in the Age of Deep Learning</RefTitle>
<RefYear>2020</RefYear>
<RefJournal>Yearb Med Inform</RefJournal>
<RefPage>208-20</RefPage>
<RefTotal>Hahn U, Oleynik M. Medical Information Extraction in the Age of Deep Learning. Yearb Med Inform. 2020 Aug;29(1):208-20. DOI: 10.1055/s-0040-1702001</RefTotal>
<RefLink>https://doi.org/10.1055/s-0040-1702001</RefLink>
</Reference>
<Reference refNo="6">
<RefAuthor>Schulz S</RefAuthor>
<RefAuthor>Daumke P</RefAuthor>
<RefAuthor>Romacker M</RefAuthor>
<RefAuthor>López-García P</RefAuthor>
<RefTitle>Representing oncology in datasets: Standard or custom biomedical terminology? Informatics in Medicine Unlocked</RefTitle>
<RefYear>2019</RefYear>
<RefTotal>Schulz S, Daumke P, Romacker M, López-García P. Representing oncology in datasets: Standard or custom biomedical terminology? Informatics in Medicine Unlocked. 2019; 15:100186. DOI: 10.1016/j.imu.2019.100186</RefTotal>
<RefLink>https://doi.org/10.1016/j.imu.2019.100186</RefLink>
</Reference>
<Reference refNo="7">
<RefAuthor>Schulz S</RefAuthor>
<RefAuthor>Jansen L</RefAuthor>
<RefTitle>Formal ontologies in biomedical knowledge representation</RefTitle>
<RefYear>2013</RefYear>
<RefJournal>Yearb Med Inform</RefJournal>
<RefPage>132-46</RefPage>
<RefTotal>Schulz S, Jansen L. Formal ontologies in biomedical knowledge representation. Yearb Med Inform. 2013;8:132-46. DOI: 10.1055/s-0038-1638845</RefTotal>
<RefLink>https://doi.org/10.1055/s-0038-1638845</RefLink>
</Reference>
<Reference refNo="10">
<RefAuthor>Oleynik M</RefAuthor>
<RefAuthor>Kugic A</RefAuthor>
<RefAuthor>Kasáč Z</RefAuthor>
<RefAuthor>Kreuzthaler M</RefAuthor>
<RefTitle>Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification</RefTitle>
<RefYear>2019</RefYear>
<RefJournal>J Am Med Inform Assoc</RefJournal>
<RefPage>1247-54</RefPage>
<RefTotal>Oleynik M, Kugic A, Kasáč Z, Kreuzthaler M. Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification. J Am Med Inform Assoc. 2019 Nov;26(11):1247-54. DOI: 10.1093/jamia/ocz149</RefTotal>
<RefLink>https://doi.org/10.1093/jamia/ocz149</RefLink>
</Reference>
<Reference refNo="20">
<RefAuthor>Ferrucci D</RefAuthor>
<RefAuthor>Lally A</RefAuthor>
<RefTitle>UIMA: an architectural approach to unstructured information processing in the corporate research environment</RefTitle>
<RefYear>2004</RefYear>
<RefJournal>Nat Lang Eng</RefJournal>
<RefPage>327-48</RefPage>
<RefTotal>Ferrucci D, Lally A. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng. 2004;10(3-4):327-48. DOI: 10.1017/S1351324904003523</RefTotal>
<RefLink>https://doi.org/10.1017/S1351324904003523</RefLink>
</Reference>
<Reference refNo="11">
<RefAuthor>Pokora RM</RefAuthor>
<RefAuthor>Le Cornet L</RefAuthor>
<RefAuthor>Daumke P</RefAuthor>
<RefAuthor>Mildenberger P</RefAuthor>
<RefAuthor>Zeeb H</RefAuthor>
<RefAuthor>Blettner M</RefAuthor>
<RefTitle>Validierung von semantischen Analysen von unstrukturierten medizinischen Daten für Forschungszwecke Validation of Semantic Analyses of Unstructured Medical Data for Research Purposes</RefTitle>
<RefYear>2020</RefYear>
<RefJournal>Gesundheitswesen</RefJournal>
<RefPage>S158-S164</RefPage>
<RefTotal>Pokora RM, Le Cornet L, Daumke P, Mildenberger P, Zeeb H, Blettner M. Validierung von semantischen Analysen von unstrukturierten medizinischen Daten für Forschungszwecke Validation of Semantic Analyses of Unstructured Medical Data for Research Purposes. Gesundheitswesen. 2020 Mar;82(S 02):S158-S164. DOI: 10.1055/a-1007-8540</RefTotal>
<RefLink>https://doi.org/10.1055/a-1007-8540</RefLink>
</Reference>
<Reference refNo="12">
<RefAuthor>Averbis</RefAuthor>
<RefTitle></RefTitle>
<RefYear></RefYear>
<RefBookTitle>Health discovery</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Averbis. Health discovery. [last accessed 2020 Nov 14]. Available from: https://averbis.com/health-discovery/</RefTotal>
<RefLink>https://averbis.com/health-discovery/</RefLink>
</Reference>
<Reference refNo="21">
<RefAuthor>Kluegl P</RefAuthor>
<RefAuthor>Toepfer M</RefAuthor>
<RefAuthor>Beck PD</RefAuthor>
<RefAuthor>Fette G</RefAuthor>
<RefAuthor>Puppe F</RefAuthor>
<RefTitle>UIMA Ruta: Rapid development of rule-based information extraction applications</RefTitle>
<RefYear>2016</RefYear>
<RefJournal>Nat Lang Eng</RefJournal>
<RefPage>1-40</RefPage>
<RefTotal>Kluegl P, Toepfer M, Beck PD, Fette G, Puppe F. UIMA Ruta: Rapid development of rule-based information extraction applications. Nat Lang Eng. 2016;22(1):1-40. DOI: 10.1017/S1351324914000114</RefTotal>
<RefLink>https://doi.org/10.1017/S1351324914000114</RefLink>
</Reference>
<Reference refNo="23">
<RefAuthor>Kim Y</RefAuthor>
<RefTitle>Convolutional Neural Networks for Sentence Classification</RefTitle>
<RefYear></RefYear>
<RefBookTitle>Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing; 2014 Oct 25-29; Doha, Quatar</RefBookTitle>
<RefPage>1746–51</RefPage>
<RefTotal>Kim Y. Convolutional Neural Networks for Sentence Classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing; 2014 Oct 25-29; Doha, Quatar. p. 1746–51. DOI: 10.3115/v1/D14-1181</RefTotal>
<RefLink>https://doi.org/10.3115/v1/D14-1181</RefLink>
</Reference>
<Reference refNo="24">
<RefAuthor>Abadi M</RefAuthor>
<RefAuthor>Barham P</RefAuthor>
<RefAuthor>Chen J</RefAuthor>
<RefAuthor>Chen Z</RefAuthor>
<RefAuthor>Davis A</RefAuthor>
<RefAuthor>Dean J</RefAuthor>
<RefAuthor>Kudlur M</RefAuthor>
<RefTitle>Tensorflow: A system for large-scale machine learning</RefTitle>
<RefYear></RefYear>
<RefBookTitle>12th USENIX symposium on operating systems design and implementation (OSDI 16); 2016 Nov 2–4; Savannah, GA, USA</RefBookTitle>
<RefPage>265-83</RefPage>
<RefTotal>Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Kudlur M. Tensorflow: A system for large-scale machine learning. In: 12th USENIX symposium on operating systems design and implementation (OSDI 16); 2016 Nov 2–4; Savannah, GA, USA. p. 265-83.</RefTotal>
</Reference>
<Reference refNo="27">
<RefAuthor>Gump V</RefAuthor>
<RefAuthor>Henß H</RefAuthor>
<RefAuthor> Universitätsklinikum Freiburg</RefAuthor>
<RefTitle></RefTitle>
<RefYear>2014</RefYear>
<RefBookTitle>Klinisches Krebsregister. Kodierhilfe. Kolonkarzinom</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Gump V, Henß H; Universitätsklinikum Freiburg. Klinisches Krebsregister. Kodierhilfe. Kolonkarzinom. 2014 [last accessed 2020 Nov 14]. Available from: https://www.uniklinik-freiburg.de/fileadmin/mediapool/09_zentren/cccf/pdf/cccf_kkr_kodierhilfe_kolontumor.pdf</RefTotal>
<RefLink>https://www.uniklinik-freiburg.de/fileadmin/mediapool/09_zentren/cccf/pdf/cccf_kkr_kodierhilfe_kolontumor.pdf</RefLink>
</Reference>
<Reference refNo="2">
<RefAuthor>Piñeros M</RefAuthor>
<RefAuthor>Parkin DM</RefAuthor>
<RefAuthor>Ward K</RefAuthor>
<RefAuthor>Chokunonga E</RefAuthor>
<RefAuthor>Ervik M</RefAuthor>
<RefAuthor>Farrugia H</RefAuthor>
<RefAuthor>Gospodarowicz M</RefAuthor>
<RefAuthor>O'Sullivan B</RefAuthor>
<RefAuthor>Soerjomataram I</RefAuthor>
<RefAuthor>Swaminathan R</RefAuthor>
<RefAuthor>Znaor A</RefAuthor>
<RefAuthor>Bray F</RefAuthor>
<RefAuthor>Brierley J</RefAuthor>
<RefTitle>Essential TNM: a registry tool to reduce gaps in cancer staging information</RefTitle>
<RefYear>2019</RefYear>
<RefJournal>Lancet Oncol</RefJournal>
<RefPage>e103-e111</RefPage>
<RefTotal>Piñeros M, Parkin DM, Ward K, Chokunonga E, Ervik M, Farrugia H, Gospodarowicz M, O'Sullivan B, Soerjomataram I, Swaminathan R, Znaor A, Bray F, Brierley J. Essential TNM: a registry tool to reduce gaps in cancer staging information. Lancet Oncol. 2019 Feb;20(2):e103-e111. DOI: 10.1016/S1470-2045(18)30897-0</RefTotal>
<RefLink>https://doi.org/10.1016/S1470-2045(18)30897-0</RefLink>
</Reference>
<Reference refNo="3">
<RefAuthor>Tyczynski JE</RefAuthor>
<RefAuthor>Démaret E</RefAuthor>
<RefAuthor>Maxwell Parkin D</RefAuthor>
<RefTitle></RefTitle>
<RefYear>2003</RefYear>
<RefBookTitle>Standards and Guidelines for Cancer Registration in Europe. The ENCR Recommendations Volume 1</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Tyczynski JE, Démaret E, Maxwell Parkin D, editors. Standards and Guidelines for Cancer Registration in Europe. The ENCR Recommendations Volume 1. Lyon: IARC Press; 2003. (IARC Technical Publication; 40).</RefTotal>
</Reference>
<Reference refNo="28">
<RefAuthor>Reiner A</RefAuthor>
<RefAuthor></RefAuthor>
<RefTitle>Pathologie des Mammakarzinoms</RefTitle>
<RefYear>1993</RefYear>
<RefBookTitle>Consensus-Bericht Mammakarzinom</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Reiner A, et al. Pathologie des Mammakarzinoms. In: Smola MG; Arbeitsgemeinschaft für Chirurgische Onkologie der Österreichischen Gesellschaft für Chirurgische Onkologie, editors. Consensus-Bericht Mammakarzinom. Graz; 1993 [last accessed 2020 Nov 14]. Available from: http://www.aco-asso.at/publikationen/aco-asso-consensusberichte/consensus-bericht-mammakarzinom/4-pathologie-des-mammakarzinoms/</RefTotal>
<RefLink>http://www.aco-asso.at/publikationen/aco-asso-consensusberichte/consensus-bericht-mammakarzinom/4-pathologie-des-mammakarzinoms/</RefLink>
</Reference>
<Reference refNo="8">
<RefAuthor>Millar J</RefAuthor>
<RefTitle>The Need for a Global Language - SNOMED CT Introduction</RefTitle>
<RefYear>2016</RefYear>
<RefJournal>Stud Health Technol Inform</RefJournal>
<RefPage>683-5</RefPage>
<RefTotal>Millar J. The Need for a Global Language - SNOMED CT Introduction. Stud Health Technol Inform. 2016;225:683-5.</RefTotal>
</Reference>
<Reference refNo="9">
<RefAuthor>Fragoso G</RefAuthor>
<RefAuthor>de Coronado S</RefAuthor>
<RefAuthor>Haber M</RefAuthor>
<RefAuthor>Hartel F</RefAuthor>
<RefAuthor>Wright L</RefAuthor>
<RefTitle>Overview and utilization of the NCI thesaurus</RefTitle>
<RefYear>2004</RefYear>
<RefJournal>Comp Funct Genomics</RefJournal>
<RefPage>648-54</RefPage>
<RefTotal>Fragoso G, de Coronado S, Haber M, Hartel F, Wright L. Overview and utilization of the NCI thesaurus. Comp Funct Genomics. 2004;5(8):648-54. DOI: 10.1002/cfg.445</RefTotal>
<RefLink>https://doi.org/10.1002/cfg.445</RefLink>
</Reference>
<Reference refNo="18">
<RefAuthor>Daumke P</RefAuthor>
<RefAuthor>Simon K</RefAuthor>
<RefAuthor>Paetzold J</RefAuthor>
<RefAuthor>Marwede D</RefAuthor>
<RefAuthor>Kotter E</RefAuthor>
<RefTitle>Data-Mining in radiologischen Befundtexten</RefTitle>
<RefYear>2010</RefYear>
<RefJournal>Rofo</RefJournal>
<RefPage></RefPage>
<RefTotal>Daumke P, Simon K, Paetzold J, Marwede D, Kotter E. Data-Mining in radiologischen Befundtexten. Rofo. 2010;182 - WS117_3. DOI: 10.1055/s-0030-1252462</RefTotal>
<RefLink>https://doi.org/10.1055/s-0030-1252462</RefLink>
</Reference>
<Reference refNo="19">
<RefAuthor>Seuss H</RefAuthor>
<RefAuthor>Dankerl P</RefAuthor>
<RefAuthor>Ihle M</RefAuthor>
<RefAuthor>Grandjean A</RefAuthor>
<RefAuthor>Hammon R</RefAuthor>
<RefAuthor>Kaestle N</RefAuthor>
<RefAuthor>Fasching PA</RefAuthor>
<RefAuthor>Maier C</RefAuthor>
<RefAuthor>Christoph J</RefAuthor>
<RefAuthor>Sedlmayr M</RefAuthor>
<RefAuthor>Uder M</RefAuthor>
<RefAuthor>Cavallaro A</RefAuthor>
<RefAuthor>Hammon M</RefAuthor>
<RefTitle>Semi-automatische Deidentifizierung von deutschsprachigen medizinischen Berichten mit vertraulichem Inhalt für Big Data Analysen Semi-automated De-identification of German Content Sensitive Reports for Big Data Analytics</RefTitle>
<RefYear>2017</RefYear>
<RefJournal>Rofo</RefJournal>
<RefPage>e1</RefPage>
<RefTotal>Seuss H, Dankerl P, Ihle M, Grandjean A, Hammon R, Kaestle N, Fasching PA, Maier C, Christoph J, Sedlmayr M, Uder M, Cavallaro A, Hammon M. Semi-automatische Deidentifizierung von deutschsprachigen medizinischen Berichten mit vertraulichem Inhalt für Big Data Analysen Semi-automated De-identification of German Content Sensitive Reports for Big Data Analytics. Rofo. 2017 Jul;189(7):e1. DOI: 10.1055/s-0035-1567202</RefTotal>
<RefLink>https://doi.org/10.1055/s-0035-1567202</RefLink>
</Reference>
<Reference refNo="26">
<RefAuthor>Qiu JX</RefAuthor>
<RefAuthor>Yoon HJ</RefAuthor>
<RefAuthor>Fearn PA</RefAuthor>
<RefAuthor>Tourassi GD</RefAuthor>
<RefTitle>Deep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports</RefTitle>
<RefYear>2018</RefYear>
<RefJournal>IEEE J Biomed Health Inform</RefJournal>
<RefPage>244-51</RefPage>
<RefTotal>Qiu JX, Yoon HJ, Fearn PA, Tourassi GD. Deep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports. IEEE J Biomed Health Inform. 2018 Jan;22(1):244-51. DOI: 10.1109/JBHI.2017.2700722</RefTotal>
<RefLink>https://doi.org/10.1109/JBHI.2017.2700722</RefLink>
</Reference>
<Reference refNo="16">
<RefAuthor>Catarino C</RefAuthor>
<RefAuthor>Grandjean A</RefAuthor>
<RefAuthor>Doss S</RefAuthor>
<RefAuthor>Mücke M</RefAuthor>
<RefAuthor>Tunc S</RefAuthor>
<RefAuthor>Schmidt K</RefAuthor>
<RefAuthor>Schmidt J</RefAuthor>
<RefAuthor>Young P</RefAuthor>
<RefAuthor>Bäumer T</RefAuthor>
<RefAuthor>Kornblum C</RefAuthor>
<RefAuthor>Endres M</RefAuthor>
<RefAuthor>Daumke P</RefAuthor>
<RefAuthor>Klopstock T</RefAuthor>
<RefAuthor>Schoser B</RefAuthor>
<RefTitle>mineRARE: Semantic text-mining of electronic medical records as diagnostic decision support tool to search for rare neurologic diseases such as Pompe disease, Fabry disease and Niemann-Pick type C disease</RefTitle>
<RefYear>2017</RefYear>
<RefJournal>Eur J Neurol</RefJournal>
<RefPage>75</RefPage>
<RefTotal>Catarino C, Grandjean A, Doss S, Mücke M, Tunc S, Schmidt K, Schmidt J, Young P, Bäumer T, Kornblum C, Endres M, Daumke P, Klopstock T, Schoser B. mineRARE: Semantic text-mining of electronic medical records as diagnostic decision support tool to search for rare neurologic diseases such as Pompe disease, Fabry disease and Niemann-Pick type C disease. Eur J Neurol. 2017;24:75 </RefTotal>
</Reference>
<Reference refNo="14">
<RefAuthor>Meta It</RefAuthor>
<RefTitle></RefTitle>
<RefYear></RefYear>
<RefBookTitle>Metakis</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Meta It. Metakis. [last accessed 2020 Nov 14]. Available from: https://metait.de/metakis</RefTotal>
<RefLink>https://metait.de/metakis</RefLink>
</Reference>
<Reference refNo="13">
<RefAuthor>TriNetX</RefAuthor>
<RefTitle></RefTitle>
<RefYear></RefYear>
<RefBookTitle>Natural Language Processing. Extract Clinical Facts from Physician Notes and Clinical Reports</RefBookTitle>
<RefPage></RefPage>
<RefTotal>TriNetX. Natural Language Processing. Extract Clinical Facts from Physician Notes and Clinical Reports. [last accessed 2020 Nov 14]. Available from: https://www.trinetx.com/nlp/</RefTotal>
<RefLink>https://www.trinetx.com/nlp/</RefLink>
</Reference>
<Reference refNo="15">
<RefAuthor>CompuGroup Medical Deutschland AG</RefAuthor>
<RefTitle></RefTitle>
<RefYear></RefYear>
<RefBookTitle>Ärzte erfassen 28 Stunden pro Woche Arztbriefe – Innovatives Werkzeug spart Zeit dank Volltextsuche und Textanalyse</RefBookTitle>
<RefPage></RefPage>
<RefTotal>CompuGroup Medical Deutschland AG. Ärzte erfassen 28 Stunden pro Woche Arztbriefe – Innovatives Werkzeug spart Zeit dank Volltextsuche und Textanalyse. [last accessed 2020 Nov 14]. Available from: https://www.cgm.com/de/ueber_uns_de/news_de/presse_de/presse_details_92416.de.jsp</RefTotal>
<RefLink>https://www.cgm.com/de/ueber_uns_de/news_de/presse_de/presse_details_92416.de.jsp</RefLink>
</Reference>
<Reference refNo="17">
<RefAuthor>Schober D</RefAuthor>
<RefAuthor>Choquet R</RefAuthor>
<RefAuthor>Depraetere K</RefAuthor>
<RefAuthor>Enders F</RefAuthor>
<RefAuthor>Daumke P</RefAuthor>
<RefAuthor>Jaulent MC</RefAuthor>
<RefAuthor>Teodoro D</RefAuthor>
<RefAuthor>Pasche E</RefAuthor>
<RefAuthor>Lovis C</RefAuthor>
<RefAuthor>Boeker M</RefAuthor>
<RefTitle>DebugIT: Ontology-mediated Layered Data Integration for Real-time Antibiotics Resistance Surveillance</RefTitle>
<RefYear></RefYear>
<RefBookTitle>Proceedings of the 7th International Workshop on Semantic Web Applications and Tools for Life Sciences (SWAT4LS 2014); 2014 Dec 9-11; Berlin, Germany</RefBookTitle>
<RefPage></RefPage>
<RefTotal>Schober D, Choquet R, Depraetere K, Enders F, Daumke P, Jaulent MC, Teodoro D, Pasche E, Lovis C, Boeker M. DebugIT: Ontology-mediated Layered Data Integration for Real-time Antibiotics Resistance Surveillance. In: Paschke A, Burger A, Romano P, Marshall MS, Splendiani A, editors. Proceedings of the 7th International Workshop on Semantic Web Applications and Tools for Life Sciences (SWAT4LS 2014); 2014 Dec 9-11; Berlin, Germany. Available from: http://ceur-ws.org/Vol-1320/paper_22.pdf</RefTotal>
<RefLink>http://ceur-ws.org/Vol-1320/paper_22.pdf</RefLink>
</Reference>
<Reference refNo="22">
<RefAuthor>Leopold E</RefAuthor>
<RefAuthor>Kindermann J</RefAuthor>
<RefTitle>Text Categorization with Support Vector Machines.How to Represent Texts in Input Space?</RefTitle>
<RefYear>2002</RefYear>
<RefJournal>Mach Learn</RefJournal>
<RefPage>423–4</RefPage>
<RefTotal>Leopold E, Kindermann J. Text Categorization with Support Vector Machines. How to Represent Texts in Input Space? Mach Learn. 2002;46:423–4. DOI: 10.1023/A:1012491419635</RefTotal>
<RefLink>https://doi.org/10.1023/A:1012491419635</RefLink>
</Reference>
<Reference refNo="29">
<RefAuthor>Bojanowski P</RefAuthor>
<RefAuthor>Grave E</RefAuthor>
<RefAuthor>Joulin A</RefAuthor>
<RefAuthor>Mikolov T</RefAuthor>
<RefTitle>Enriching Word Vectors with Subword Information</RefTitle>
<RefYear>2017</RefYear>
<RefJournal>Trans Assoc Comput Linguist</RefJournal>
<RefPage>135–46</RefPage>
<RefTotal>Bojanowski P, Grave E, Joulin A, Mikolov T. Enriching Word Vectors with Subword Information. Trans Assoc Comput Linguist. 2017;5:135–46. DOI: 10.1162/tacl_a_00051</RefTotal>
<RefLink>https://doi.org/10.1162/tacl_a_00051</RefLink>
</Reference>
<Reference refNo="30">
<RefAuthor>Falkner S</RefAuthor>
<RefAuthor>Klein A</RefAuthor>
<RefAuthor>Hutter F</RefAuthor>
<RefTitle>BOHB: Robust and Efficient Hyperparameter Optimization at Scale</RefTitle>
<RefYear></RefYear>
<RefBookTitle>Proceedings of the 35th International Conference on Machine Learning; 2018 Jul 10-15; Stockholm, Sweden</RefBookTitle>
<RefPage>1437–46</RefPage>
<RefTotal>Falkner S, Klein A, Hutter F. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In: Proceedings of the 35th International Conference on Machine Learning; 2018 Jul 10-15; Stockholm, Sweden. (Proceedings of Machine Learning Research; 80). p. 1437–46.</RefTotal>
</Reference>
</References>
<Media>
<Tables>
<Table format="png">
<MediaNo>1</MediaNo>
<MediaID>1</MediaID>
<Caption><Pgraph><Mark1>Table 1: Synopsis of cancer registries in three German federal states</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>2</MediaNo>
<MediaID>2</MediaID>
<Caption><Pgraph><Mark1>Table 2: Document sets and properties used for this study</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>3</MediaNo>
<MediaID>3</MediaID>
<Caption><Pgraph><Mark1>Table 3: Fields of structured datasets used for comparison with automated text analysis</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>4</MediaNo>
<MediaID>4</MediaID>
<Caption><Pgraph><Mark1>Table 4: Pathology pipeline</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>5</MediaNo>
<MediaID>5</MediaID>
<Caption><Pgraph><Mark1>Table 5: Results of a five-fold cross evaluation for cancer type classification using SVMs</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>6</MediaNo>
<MediaID>6</MediaID>
<Caption><Pgraph><Mark1>Table 6: Comparison between SVM and CNN on tumour localisation and morphology classification</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>7</MediaNo>
<MediaID>7</MediaID>
<Caption><Pgraph><Mark1>Table 7: Evaluation results by feature for the BW dataset. The diagnosis values (ICD-10-GM) are derived from the tumour localisation and morphology ICD-O codes.</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>8</MediaNo>
<MediaID>8</MediaID>
<Caption><Pgraph><Mark1>Table 8: Top 10 deviations (BW dataset)</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>9</MediaNo>
<MediaID>9</MediaID>
<Caption><Pgraph><Mark1>Table 9: Deviations by categories (BW dataset)</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>10</MediaNo>
<MediaID>10</MediaID>
<Caption><Pgraph><Mark1>Table 10: Evaluation results by feature (RP dataset). The diagnosis values (ICD-10-GM) are derived from the tumour localisation and morphology ICD-O codes.</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>11</MediaNo>
<MediaID>11</MediaID>
<Caption><Pgraph><Mark1>Table 11: Top 10 deviations (RP dataset)</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>12</MediaNo>
<MediaID>12</MediaID>
<Caption><Pgraph><Mark1>Table 12: Deviations by categories (RP dataset)</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>13</MediaNo>
<MediaID>13</MediaID>
<Caption><Pgraph><Mark1>Table 13: Evaluation results by feature for the LS dataset. The diagnosis values (ICD-10-GM) are derived from the tumour localisation and morphology ICD-O codes.</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>14</MediaNo>
<MediaID>14</MediaID>
<Caption><Pgraph><Mark1>Table 14: Top 10 deviations (LS dataset)</Mark1></Pgraph></Caption>
</Table>
<Table format="png">
<MediaNo>15</MediaNo>
<MediaID>15</MediaID>
<Caption><Pgraph><Mark1>Table 15: Deviations by categories (LS dataset)</Mark1></Pgraph></Caption>
</Table>
<NoOfTables>15</NoOfTables>
</Tables>
<Figures>
<Figure format="png" height="336" width="939">
<MediaNo>1</MediaNo>
<MediaID>1</MediaID>
<Caption><Pgraph><Mark1>Figure 1: Annotations of the pathology pipeline, displayed in the AHD annotation editor</Mark1></Pgraph></Caption>
</Figure>
<Figure format="png" height="383" width="860">
<MediaNo>2</MediaNo>
<MediaID>2</MediaID>
<Caption><Pgraph><Mark1>Figure 2: Approach to determine the final output fields of the pathology pipeline</Mark1></Pgraph></Caption>
</Figure>
<Figure format="png" height="365" width="970">
<MediaNo>3</MediaNo>
<MediaID>3</MediaID>
<Caption><Pgraph><Mark1>Figure 3: Overview of evaluation results. Dark blue BW (Baden Württemberg state), medium blue LS (Lower Saxony), light blue RP (Rhineland Palatinate)</Mark1></Pgraph></Caption>
</Figure>
<NoOfPictures>3</NoOfPictures>
</Figures>
<InlineFigures>
<NoOfPictures>0</NoOfPictures>
</InlineFigures>
<Attachments>
<NoOfAttachments>0</NoOfAttachments>
</Attachments>
</Media>
</OrigData>
</GmsArticle>]]></plaintext></textgroup></pgraph></textblock></origdata></gmsarticle>