Academic Journal
View in EDS
PDF Full Text
HTML Full Text

Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides

Bibliographic Details
Title: Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides
Authors: Sara P. Oliveira, João Ribeiro Pinto, Tiago Gonçalves, Rita Canas-Marques, Maria-João Cardoso, Hélder P. Oliveira, Jaime S. Cardoso
Source: Applied Sciences, Vol 10, Iss 14, p 4728 (2020)
Publisher Information: MDPI AG, 2020.
Publication Year: 2020
Collection: LCC:Technology
LCC:Engineering (General). Civil engineering (General)
LCC:Biology (General)
LCC:Physics
LCC:Chemistry
Subject Terms: weakly-supervised learning, HER2, breast cancer, Technology, Engineering (General). Civil engineering (General), TA1-2040, Biology (General), QH301-705.5, Physics, QC1-999, Chemistry, QD1-999
More Details: Human epidermal growth factor receptor 2 (HER2) evaluation commonly requires immunohistochemistry (IHC) tests on breast cancer tissue, in addition to the standard haematoxylin and eosin (H&E) staining tests. Additional costs and time spent on further testing might be avoided if HER2 overexpression could be effectively inferred from H&E stained slides, as a preliminary indication of the IHC result. In this paper, we propose the first method that aims to achieve this goal. The proposed method is based on multiple instance learning (MIL), using a convolutional neural network (CNN) that separately processes H&E stained slide tiles and outputs an IHC label. This CNN is pretrained on IHC stained slide tiles but does not use these data during inference/testing. H&E tiles are extracted from invasive tumour areas segmented with the HASHI algorithm. The individual tile labels are then combined to obtain a single label for the whole slide. The network was trained on slides from the HER2 Scoring Contest dataset (HER2SC) and tested on two disjoint subsets of slides from the HER2SC database and the TCGA-TCIA-BRCA (BRCA) collection. The proposed method attained 83.3 % classification accuracy on the HER2SC test set and 53.8 % on the BRCA test set. Although further efforts should be devoted to achieving improved performance, the obtained results are promising, suggesting that it is possible to perform HER2 overexpression classification on H&E stained tissue slides.
Document Type: article
File Description: electronic resource
Language: English
ISSN: 2076-3417
Relation: https://www.mdpi.com/2076-3417/10/14/4728; https://doaj.org/toc/2076-3417
DOI: 10.3390/app10144728
Access URL: https://doaj.org/article/c92b78d70228495793cbff626dc0ee9e
Accession Number: edsdoj.92b78d70228495793cbff626dc0ee9e
Database: Directory of Open Access Journals
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHjPtM4BHU3ZchRwgzYmadcigk49r9CVlbU7V5F6lgH7WwFnjeDv4UA6S0ZfFz62HgQ5AAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDMG31pJk-JoTIEnhegIBEICBmkec2Wj7npHw9k-vrM5rm-xrNJPq_-zWUi2tiOYJGtgPZn44e4kJgDaKN_G8Ud8snC9Mh9YjXydkHbEeJQo481OXPguPHB-qmgDIDzNIPy_MeWHEyQmSKtfxmbjqx_n5mupEgRyPCmlEE7Dz6E30sa2TvepNNWEUsrisNLsBJ7PfSt1gOWaIkzysiOkbQ_lVMPTxEVr8LDaStl8=
Text:
  Availability: 1
  Value: <anid>AN0144773759;[fdtu]15jul.20;2020Jul28.05:55;v2.2.500</anid> <title id="AN0144773759-1">Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides </title> <p>Featured Application: This work finds its key application in medical diagnosis and prognosis. It paves the way to robust automatic HER2 classification using only H&E slides. This approach possibly avoids the additional costs and time spent IHC testing, providing an indication of the IHC result. Human epidermal growth factor receptor 2 (HER2) evaluation commonly requires immunohistochemistry (IHC) tests on breast cancer tissue, in addition to the standard haematoxylin and eosin (H&E) staining tests. Additional costs and time spent on further testing might be avoided if HER2 overexpression could be effectively inferred from H&E stained slides, as a preliminary indication of the IHC result. In this paper, we propose the first method that aims to achieve this goal. The proposed method is based on multiple instance learning (MIL), using a convolutional neural network (CNN) that separately processes H&E stained slide tiles and outputs an IHC label. This CNN is pretrained on IHC stained slide tiles but does not use these data during inference/testing. H&E tiles are extracted from invasive tumour areas segmented with the HASHI algorithm. The individual tile labels are then combined to obtain a single label for the whole slide. The network was trained on slides from the HER2 Scoring Contest dataset (HER2SC) and tested on two disjoint subsets of slides from the HER2SC database and the TCGA-TCIA-BRCA (BRCA) collection. The proposed method attained 83.3 % classification accuracy on the HER2SC test set and 53.8 % on the BRCA test set. Although further efforts should be devoted to achieving improved performance, the obtained results are promising, suggesting that it is possible to perform HER2 overexpression classification on H&E stained tissue slides.</p> <p>Keywords: weakly-supervised learning; HER2; breast cancer</p> <hd id="AN0144773759-2">1. Introduction</hd> <p>Breast cancer (BCa) is the most commonly diagnosed cancer and the leading cause of cancer-related deaths among women worldwide. However, over the most recent years, despite the increasing incidence trends, the mortality rate has significantly decreased. Among other factors, this improvement results from better treatment strategies that can be delineated from the assessment of histopathological characteristics [[<reflink idref="bib1" id="ref1">1</reflink>]].</p> <p>The analysis of tissue sections of cancer specimens (Figure 1) obtained by preoperative biopsy, commonly starts with haematoxylin and eosin (H&E) staining, which is usually followed by immunohistochemistry (IHC), a more advanced staining technique, used to highlight the presence of specific protein receptors [[<reflink idref="bib3" id="ref2">3</reflink>]]. In fact, according to the current guidelines [[<reflink idref="bib4" id="ref3">4</reflink>]], Human epidermal growth factor receptor 2 (HER2) quantification must be routinely tested in all patients with invasive BCa, recurrence cases, and metastatic tumours. The overexpression of this receptor is observed in 10–20% [[<reflink idref="bib4" id="ref4">4</reflink>]] of BCa cases and has been associated with aggressive clinical behaviour and poor prognosis [[<reflink idref="bib5" id="ref5">5</reflink>]]. However, patients diagnosed with HER2-positive BCa have a better response to targeted therapies and consequent improvements in healing and overall survival, which emphasizes the importance of an accurate evaluation of HER2 status [[<reflink idref="bib5" id="ref6">5</reflink>]].</p> <p>The current guidelines [[<reflink idref="bib7" id="ref7">7</reflink>]], revised by the American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP), in 2018, indicate the following scoring criteria for HER2 IHC:</p> <p></p> <ulist> <item> – IHC 0+: no staining or incomplete, barely perceptible membrane staining in 10% of tumour cells or less;</item> <p></p> <item> – IHC 1+: incomplete, barely perceptible membrane staining in more than 10% of tumour cells;</item> <p></p> <item> – IHC 2+: weak to moderate complete membrane staining in more than 10% of tumour cells;</item> <p></p> <item> – IHC 3+: circumferential, complete, intense membrane staining in more than 10% of tumour cells.</item> </ulist> <p>Moreover, cases scoring <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>+</mo></mrow></semantics></math> </ephtml> or <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo></mrow></semantics></math> </ephtml> are classified as HER2 negative, while cases with a score of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>+</mo></mrow></semantics></math> </ephtml> are classified as HER2 positive. Cases with score <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo></mrow></semantics></math> </ephtml> are classified as equivocal and are further assessed by in situ hybridization (ISH), to test for gene amplification (see Figure 1). In these cases, the HER2 status is given by the ISH result [[<reflink idref="bib7" id="ref8">7</reflink>]].</p> <p>At the moment, besides very well differentiated tumours, with low nuclear/cytoplasm area ratio, which typically are hormonal driven and therefore generally not positive for HER2, there are no morphological features on H&E slides that allow a reliable prediction of the HER2 status. Therefore, the standard procedure is to perform an additional immunohistochemical study, with additional molecular study in case of equivocal results. Despite the efficiency of IHC and ISH, the additional cost and time spent on these tests might be avoided if all the information needed to infer the HER2 status could be extracted only from H&E whole slide images (WSI), as a preliminary indication of the IHC result. However, to the extent of our knowledge, the task of predicting HER2 status on H&E stained slides has not yet been addressed in the literature, except for a recent challenge: ECDP2020's HEROHE Challenge [[<reflink idref="bib9" id="ref9">9</reflink>]].</p> <p>In this paper we propose a method using a convolutional neural network (CNN), inspired by multiple-instance learning (MIL), to automatically identify the HER2 status on BCa H&E stained slides. To deal with the sheer dimensions of the slides, tiles are extracted from the original images and separately processed by the model, which learns to aggregate the individual tile predictions into a single, image-wide label. Moreover, to introduce some prior knowledge about the morphology of tissue structures into the model, the CNN has been pre-trained with HER2 IHC stained slides (example in Figure 2a) from the HER2 Scoring Contest (HER2SC) training set [[<reflink idref="bib10" id="ref10">10</reflink>]]. The final architecture was trained with the H&E stained slides of HER2SC (example in Figure 2b) and tested with a disjoint subset of H&E stained slides (example in Figure 2c) from the TCIA-TCGA-BRCA (BRCA) collection [[<reflink idref="bib11" id="ref11">11</reflink>]]. The code is publicly available in a GitHub repository [[<reflink idref="bib13" id="ref12">13</reflink>]].</p> <hd id="AN0144773759-3">2. Related Work</hd> <p>With the advent of WSI over the last decade, a huge amount of tissue slides are routinely scanned in clinical practice, thereby increasing data availability. Consequently, and along with its important role in oncological clinical routine, more research opportunities are raised in computer-assisted imaging analysis, with this new source of "big data" [[<reflink idref="bib14" id="ref13">14</reflink>]]. In fact, due to the high-resolution and complex nature of this imaging technique, advances in image analysis are required, providing an opportunity to apply/develop more advanced image processing techniques, as well as machine and deep learning algorithms. [[<reflink idref="bib15" id="ref14">15</reflink>]].</p> <p>The analysis of digital images of digital breast pathology can be applied to tackle several clinical and pathology tasks such as, for example, mitosis detection [[<reflink idref="bib16" id="ref15">16</reflink>]], tissue type classification/ segmentation [[<reflink idref="bib18" id="ref16">18</reflink>]], cancer grading [[<reflink idref="bib21" id="ref17">21</reflink>]] or histological classification [[<reflink idref="bib22" id="ref18">22</reflink>]]. These approaches commonly use H&E stained slides and, in recent years, have been focused on applying deep learning techniques to improve the performance of the models and also to take advantage of the increasing availability of medical data.</p> <p>Besides H&E staining slides, some authors, such as Oscanoa et al. [[<reflink idref="bib24" id="ref19">24</reflink>]], Mojoy et al. [[<reflink idref="bib25" id="ref20">25</reflink>]], Jamaluddin et al. [[<reflink idref="bib26" id="ref21">26</reflink>]] have addressed diverse tasks using IHC slides. On the specific task of automatic breast cancer HER2 classification, the focus of this work, the literature is limited to the work by Rodner et al. [[<reflink idref="bib27" id="ref22">27</reflink>]], Mukundan [[<reflink idref="bib28" id="ref23">28</reflink>]], and other studies related with the 2016 HER2 Scoring Contest [[<reflink idref="bib10" id="ref24">10</reflink>]]. The common disadvantage of these prior works is requiring IHC staining images to perform HER2 classification, since this modality requires additional cost and time. In contrast, our approach aims at using the H&E staining slides as the only source of information to obtain the IHC status for HER2 overexpression.</p> <p>In order to complete the proposed task, we followed the idea of using a data source as initialization for the framework, transferring some domain knowledge to the final training. This is a recent trend that has been applied to medical imaging processing for different purposes, such as cardiac structures segmentation [[<reflink idref="bib29" id="ref25">29</reflink>]], Alzheimer disease classification [[<reflink idref="bib30" id="ref26">30</reflink>]], radiological breast lesions classification [[<reflink idref="bib31" id="ref27">31</reflink>]] and even digital pathology classification/segmentation [[<reflink idref="bib32" id="ref28">32</reflink>]].</p> <p>Despite the growing popularity of digital pathology and the increasing number of publications in this area, to the extent of our knowledge, the task of predicting HER2 status on H&E stained slides has not yet been addressed in the literature.</p> <hd id="AN0144773759-4">3. Methodology</hd> <p>The proposed method (Figure 3) comprises a convolutional neural network (CNN), which is pre-trained for the task of HER2 scoring of tiles extracted from IHC stained slides. The pre-trained parameters are then transferred to the task of HER2 status prediction on H&E staining slide tiles, to provide the network with some knowledge of the tissue structures' appearance. Individual tile scores are then combined to obtain a single label for the respective slide. The data preprocessing methodology and the implemented networks are described below.</p> <hd id="AN0144773759-5">3.1. Data Preprocessing</hd> <p></p> <hd id="AN0144773759-6">3.1.1. IHC Stained Slides</hd> <p>For the IHC stained slides of classes <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo></mrow></semantics></math> </ephtml> and <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>+</mo></mrow></semantics></math> </ephtml> , the preprocessing begins with automatic tissue segmentation with Otsu's thresholding on the saturation (S) channel of the HSV colour space, obtaining the regions with more intense staining, that correspond to the HER overexpression areas. For slides of classes <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>+</mo></mrow></semantics></math> </ephtml> and <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo></mrow></semantics></math> </ephtml> , the segmentation consists of simple removal of pixels with the greatest HSV value (V) intensity, corresponding to background pixels, which do not contain essential information to the problem. These processes, which are performed at <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>32</mn><mo>×</mo></mrow></semantics></math> </ephtml> downsampled slides, return the masks used in tile extraction.</p> <p>Tiles with size <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>256</mn><mo>×</mo><mn>256</mn></mrow></semantics></math> </ephtml> are extracted from the slide with original dimensions (without downsampling), provided they are completely within the mask region. These tiles are converted from RGB to HSL colour space, of which only the lightness (L) channel is used. Each tile inherits the class from the respective slide (examples in Figure 4a–d), turning the learning task into a weakly-supervised problem.</p> <hd id="AN0144773759-7">3.1.2. H&E Stained Slides</hd> <p>According to the ASCO/CAP guidelines for IHC evaluation, the diagnosis is performed based only the tumoral region of the slides. Hence, the preprocessing of H&E stained slides begins with an automatic invasive tissue segmentation with the HASHI method [[<reflink idref="bib18" id="ref29">18</reflink>], [<reflink idref="bib33" id="ref30">33</reflink>]] (High-throughput Adaptive Sampling for whole-slide Histopathology Image analysis), which consists of an adaptive gradient-based sampling approach that iteratively refines an initial coarse invasive BCa probability map, from CNN inference.</p> <p>The algorithm begins with a WSI as input, that is sampled in 100 tiles, each of them being classified using a CNN-trained model, to obtain the probability of invasive BCa presence. By interpolation of each tile probability, a heatmap is generated for the entire WSI. Then, the gradient of the map is calculated and used to prioritize the sampling selection on the next iteration. The process is repeated during 20 iterations [[<reflink idref="bib18" id="ref31">18</reflink>]].</p> <p>The method was implemented in the images referred by Cruz-Roa et al. [[<reflink idref="bib18" id="ref32">18</reflink>]] as the test set, using the original magnification and extracting squared <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>512</mn><mo>×</mo><mn>512</mn></mrow></semantics></math> </ephtml> tiles. Moreover, to exclude eventual small background zones included in HASHI segmentation, this mask region was intersected with the segmentation obtained using Otsu's thresholding on the saturation (S) channel of the HSV colour space.</p> <p>The final segmentation mask was then used to generate H&E tiles (example in Figure 4e), extracted and processed accordingly to the methodology described for IHC slides. The number of tiles per slide varies according to the extent of the tissue region.</p> <hd id="AN0144773759-8">3.2. CNN for IHC Tile Scoring</hd> <p>The CNN architecture (Figure 5) consists of four convolutional layers (<reflink idref="bib16" id="ref33">16</reflink>, 32, 64 and 128 filters, respectively, with ReLU activation). The first layer has <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5</mn><mo>×</mo><mn>5</mn></mrow></semantics></math> </ephtml> square kernels, while the remaining have <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>×</mo><mn>3</mn></mrow></semantics></math> </ephtml> square kernels. Each convolutional layer is followed by one pooling layer (a max-pooling function without overlap, with kernel <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>2</mn></mrow></semantics></math> </ephtml> ). The network is topped with three fully-connected layers, with 1024, 256, and 4 units, respectively. The first two have ReLU activation, while the third is followed by softmax activation for the output of probabilities for each class.</p> <hd id="AN0144773759-9">3.3. CNN for H&E Stained Slide Classification</hd> <p>The network parameters pre-trained with IHC stained slides were used as initial network weights for HER2 status classification on H&E stained slides. It is worth mentioning that IHC data is only used for the network pre-train, and not on the inference/test phase. To achieve a single prediction per tile, instead of four (as it was initially trained for on the IHC setting), a soft-argmax activation [[<reflink idref="bib34" id="ref34">34</reflink>]] replaces the softmax activation, following the equation</p> <p>(<reflink idref="bib1" id="ref35">1</reflink>) <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>soft</mi><mtext>-</mtext><mi>argmax</mi></mrow><mrow><mo>(</mo><mi>s</mi><mo>)</mo></mrow><mo>=</mo><mo>∑</mo><mi>i</mi><mi>softmax</mi><mrow><mo>(</mo><mi>β</mi><msub><mi>s</mi><mi>i</mi></msub><mo>)</mo></mrow><mi>i</mi><mo>,</mo></mrow></semantics></math> </ephtml></p> <p>where <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>β</mi></semantics></math> </ephtml> is an adjustment factor which controls the range of the probability map given by the softmax, <emph>s</emph> is the tile score array, and <emph>i</emph> is the index that corresponds to each class.</p> <p>Having a single value per tile enables the easy sorting of tiles, which is performed before the aggregation into a single HER2 label. With the HER2 scores of each tile, output by the soft-argmax activation, tiles are sorted from <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>+</mo></mrow></semantics></math> </ephtml> to <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>+</mo></mrow></semantics></math> </ephtml> . Then, the <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>15</mn><mo>%</mo></mrow></semantics></math> </ephtml> highest scores are selected to serve as input to the aggregation process. This percentage was chosen to limit the information given to the aggregation network, while still including and barely exceeding the reference <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>10</mn><mo>%</mo></mrow></semantics></math> </ephtml> of tumour area considered in the HER2 scoring guidelines.</p> <p>The score aggregation is performed by a multilayer perceptron (MLP), composed of four layers, with 256, 128, 64, and 2 neurons, respectively. All layers are followed by ReLU activation, except the last layer, which is followed by softmax activation. Since the input dimension <emph>M</emph> of the MLP is fixed (we set <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi><mo>=</mo><mn>300</mn></mrow></semantics></math> </ephtml> in our experimental analysis, to limit memory cost), for images where <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>15</mn><mo>%</mo></mrow></semantics></math> </ephtml> of the number of tiles exceeds <emph>M</emph>, we downsample to 300 using evenly distributed tile selection. In cases where <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>15</mn><mo>%</mo></mrow></semantics></math> </ephtml> of the number of tiles is lower than <emph>M</emph>, tiles are extracted with overlap, to guarantee that <emph>M</emph> tiles can be selected. The MLP will process these 300 HER2 scores and output a single HER2 status label for the respective slide.</p> <hd id="AN0144773759-10">4. Experimental Settings</hd> <p></p> <hd id="AN0144773759-11">4.1. Data</hd> <p>The dataset is composed of subsets of WSI from two public datasets: the HER2 Scoring Contest (HER2SC) training set [[<reflink idref="bib10" id="ref36">10</reflink>]] and the TCGA-TCIA-BRCA (BRCA) collection [[<reflink idref="bib11" id="ref37">11</reflink>]]. The HER2SC training set (the subset with available labelling) comprises WSI of sections of 52 cases of invasive BCa stained with both IHC and H&E (example in Figure 2a,b). From this set, all IHC and H&E stained slides were used, except 4 H&E excluded because of manual ink markings. The subset from the BRCA dataset includes 54 H&E stained WSI (example in Figure 2c). All slides have the same original resolution and are weakly annotated with HER2 status (negative/positive) and score ( <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>+</mo></mrow></semantics></math> </ephtml> , <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo></mrow></semantics></math> </ephtml> , <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo></mrow></semantics></math> </ephtml> , <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>+</mo></mrow></semantics></math> </ephtml> ), obtained from the corresponding histopathological reports.</p> <p>The IHC stained slides were manually segmented into regions of interest (ROI), using the Sedeen Viewer software [[<reflink idref="bib36" id="ref38">36</reflink>]]. However, it is noteworthy that these slides were only used for training and, thus this step is not needed for testing.</p> <p>The training and validation sets, used for model parameter tuning and optimization, have 40 and 12 IHC slides, respectively. A total of 7591 tiles per class have been extracted for training ( <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>30,364</mn></mrow></semantics></math> </ephtml> tiles total) and 624 tiles per class extracted for validation (2496 tiles total), to keep a class balance.</p> <hd id="AN0144773759-12">4.2. Training Details</hd> <p>The hyperparameters used during training were empirically set to maximize performance. The CNN model for IHC tile scoring was randomly initialized and trained using the Adaptive Moment Estimation (Adam) [[<reflink idref="bib37" id="ref39">37</reflink>]] optimizer (learning rate of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>×</mo><msup><mn>10</mn><mrow><mo>−</mo><mn>5</mn></mrow></msup></mrow></semantics></math> </ephtml> ), to minimize a cross-entropy loss function, during 200 epochs, with mini-batches of 128 tiles. The soft-argmax used a parameter <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>β</mi><mo>=</mo><mn>1000</mn></mrow></semantics></math> </ephtml> . The aggregation MLP was trained using the Adam optimizer, with learning rate of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mn>10</mn><mrow><mo>−</mo><mn>5</mn></mrow></msup></semantics></math> </ephtml> for 150 epochs and mini-batches of 1 WSI (consisting of soft-argmax scores of the respective 300 tiles), saving the best considering validation accuracy.</p> <hd id="AN0144773759-13">5. Results and Discussion</hd> <p></p> <hd id="AN0144773759-14">5.1. Individual IHC Tile Scoring Results</hd> <p>After training, the model offered <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>76.8</mn><mo>%</mo></mrow></semantics></math> </ephtml> accuracy (see Table 1). This indicates that the model was able to adequately discriminate against the IHC tiles between the four classes. This model was subsequently transferred for HER2 scoring in tiles from H&E slides.</p> <hd id="AN0144773759-15">5.2. Invasive Tumor Tissue Segmentation</hd> <p>Tiles from H&E WSI are extracted from the intersection area between the HASHI-based invasive tumour segmentation and the Otsu-based tissue segmentation. The HASHI segmentation method was trained on the BRCA data reported as the test set by Cruz-Roa et al. [[<reflink idref="bib18" id="ref40">18</reflink>]], with 179 WSI on their original magnification. The results were comparable to the original paper (see examples in Figure 2) and were further evaluated by a pathology specialist, who confirmed the adequacy of the invasive tumour segmentation results.</p> <hd id="AN0144773759-16">5.3. Slide Scoring</hd> <p>On the HER2SC test set, this method achieved an F1-score of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>86.7</mn><mo>%</mo></mrow></semantics></math> </ephtml> and a weighted accuracy of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>83.3</mn><mo>%</mo></mrow></semantics></math> </ephtml> (see Table 2). Despite the small size of this test set, the proposed method was able to correctly classify all positive WSI and only misclassify one negative sample as positive. In this context, one might consider this a desirable behaviour, as false positives are less impactful than false negatives.</p> <p>When tested on the BRCA test set, this method achieved an F1-score of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>21.5</mn><mo>%</mo></mrow></semantics></math> </ephtml> and a weighted accuracy of <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>53.8</mn><mo>%</mo></mrow></semantics></math> </ephtml> (see Table 2). The method retains the behaviour presented in HER2SC, preferring to err on the side of false positives than the alternative. On the other hand, the performance metrics on BRCA differ considerably from those obtained on HER2SC. While the method was trained on HER2SC data, which is expected similar to the test data, the WSI of the BRCA dataset presents some notable differences. These slides have a greater extent of tissue, which generates more tiles per image and impacts the distribution of the scores, which may influence the method's behaviour.</p> <p>The evaluation results in single-database (HER2SC) and cross-database (BRCA) settings show the potential of the proposed method in standard and more challenging situations. However, the method appears to be dataset-dependent: it performed much better in conditions similar to the training. This should be addressed with additional efforts regarding domain adaptation.</p> <p>The other shortcomings of the method appear to be related to the invasive tumour tissue segmentation and the individual tile scoring network, which could be improved with additional data and more accurate ground truth information. With these additional efforts, the proposed method could offer robust weakly-supervised WSI HER2 classification without IHC information.</p> <hd id="AN0144773759-17">5.4. Ablation Study</hd> <p>Considering the lack of literature methods, to perform a benchmark, an ablation study was performed to confirm the capabilities of the proposed method. Experiments were conducted without IHC individual tile scoring CNN initialization, and using alternative statistical methods for individual tile score aggregation instead of MLP (median and mean), as can be seen in Table 3 and Table 4. The results show that these alternatives are, in most settings, less adequate for the task at hand.</p> <p>It is noteworthy that the median and mean-based aggregation are followed by a conversion to binary classes ( <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>+</mo></mrow></semantics></math> </ephtml> and <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo></mrow></semantics></math> </ephtml> are considered negative, while <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo></mrow></semantics></math> </ephtml> and <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>+</mo></mrow></semantics></math> </ephtml> are considered positive), since tiles have four possible labels. According to the guidelines, <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo></mrow></semantics></math> </ephtml> cases can be either negative or positive, but in an uncertain diagnosis scenario, it is preferable to classify them as positive.</p> <hd id="AN0144773759-18">6. Conclusions</hd> <p>In this work, a framework is proposed for the weakly supervised classification of HER2 overexpression status on H&E stained BCa WSI. The proposed approach integrates a CNN trained for HER2 scoring of individual H&E stained slide tiles, initialized with the network parameters pre-trained with data from IHC stained images. The objective of this initialization is to transfer some domain knowledge to the final training. The individual scores are aggregated on a single prediction per slide, returning the HER2 status label.</p> <p>Tested with the BRCA data subset, the proposed method attained suitable performance. These preliminary results indicate that it is possible to accurately infer BCa HER2 status solely from H&E stained slides. The results of an ablation study suggest that the proposed method with MLP tile score aggregation is more promising than simpler aggregation methods (mean or median).</p> <p>Despite these results, further efforts should be devoted to performance improvements in this task. Firstly, the training of the tile HER2 scoring CNN and the aggregation MLP could be integrated into a single optimization process to achieve better performance. On the other hand, the aggregation of individual scores could use tile locations to take spatial consistency into account. Finally, the knowledge embedded in the networks through the pre-trained parameters could be better seized if input H&E tiles could be previously converted into IHC (possibly using generative adversarial models).</p> <hd id="AN0144773759-19">Figures and Tables</hd> <p>Graph: Figure 1 Schema of the process of BCa HER2 evaluation, involving H&E staining, IHC testing and, in specific cases, ISH testing. The proposed method aims to evaluate HER2 using only H&E stained slide images. Image examples were adapted from [[<reflink idref="bib8" id="ref41">8</reflink>]].</p> <p>Graph: Figure 2 Image examples from used datasets: HER2SC [[<reflink idref="bib10" id="ref42">10</reflink>]] IHC stained slides (a), HER2SC [[<reflink idref="bib10" id="ref43">10</reflink>]] H&E stained slides (b), BRCA [[<reflink idref="bib11" id="ref44">11</reflink>]] H&E stained slides (c). The tile extraction was solely done on tissue, here denoted by the delineated regions.</p> <p>Graph: Figure 3 The proposed approach for weakly-supervised HER2 status classification on BCa H&E stained slides.</p> <p>Graph: Figure 4 Tile examples extracted from IHC 0+ (a), IHC 1+ (b), IHC 2+ (c), IHC 3+ (d), H&E (e) slides. Tiles from IHC 2+ and 3+ and H&E slides were obtained by Otsu's thresholding and the remaining were obtained by simply removing the pixels with background value. The IHC tiles were obtained from slides of the HER2SC dataset [[<reflink idref="bib10" id="ref45">10</reflink>]] and the H&E tile was obtained from a slide of the BRCA dataset [[<reflink idref="bib11" id="ref46">11</reflink>]].</p> <p>Graph: Figure 5 Architecture of the implemented convolutional neural network.</p> <p>Table 1 Confusion matrix of the CNN for HER2 scoring in IHC tiles.</p> <p> <ephtml> <table><thead><tr><th align="center" style="border-bottom:solid thin;border-top:solid thin" /><th align="center" style="border-bottom:solid thin;border-top:solid thin" /><th colspan="4" align="center" style="border-bottom:solid thin;border-top:solid thin">True Class</th></tr></thead><tbody><tr><td rowspan="5" align="center" valign="middle" style="border-bottom:solid thin"><bold>Prediction</bold></td><td align="center" valign="middle" style="border-bottom:solid thin" /><td align="center" valign="middle" style="border-bottom:solid thin"><italic>0</italic></td><td align="center" valign="middle" style="border-bottom:solid thin"><italic>1</italic></td><td align="center" valign="middle" style="border-bottom:solid thin"><italic>2</italic></td><td align="center" valign="middle" style="border-bottom:solid thin"><italic>3</italic></td></tr><tr><td align="center" valign="middle" style="border-bottom:solid thin"><italic>0</italic></td><td align="center" valign="middle" style="background:#D8D8D8;border-bottom:solid thin">490</td><td align="center" valign="middle" style="border-bottom:solid thin">132</td><td align="center" valign="middle" style="border-bottom:solid thin">2</td><td align="center" valign="middle" style="border-bottom:solid thin">0</td></tr><tr><td align="center" valign="middle" style="border-bottom:solid thin"><italic>1</italic></td><td align="center" valign="middle" style="border-bottom:solid thin">176</td><td align="center" valign="middle" style="background:#D8D8D8;border-bottom:solid thin">384</td><td align="center" valign="middle" style="border-bottom:solid thin">64</td><td align="center" valign="middle" style="border-bottom:solid thin">0</td></tr><tr><td align="center" valign="middle" style="border-bottom:solid thin"><italic>2</italic></td><td align="center" valign="middle" style="border-bottom:solid thin">45</td><td align="center" valign="middle" style="border-bottom:solid thin">159</td><td align="center" valign="middle" style="background:#D8D8D8;border-bottom:solid thin">419</td><td align="center" valign="middle" style="border-bottom:solid thin">1</td></tr><tr><td align="center" valign="middle" style="border-bottom:solid thin"><italic>3</italic></td><td align="center" valign="middle" style="border-bottom:solid thin">0</td><td align="center" valign="middle" style="border-bottom:solid thin">0</td><td align="center" valign="middle" style="border-bottom:solid thin">1</td><td align="center" valign="middle" style="background:#D8D8D8;border-bottom:solid thin">623</td></tr></tbody></table> </ephtml> </p> <p>Table 2 H&E HER2 status classification results of the proposed method on the HER2SC and BRCA test sets.</p> <p> <ephtml> <table><thead><tr><th align="left" style="border-bottom:solid thin;border-top:solid thin" /><th align="center" style="border-bottom:solid thin;border-top:solid thin">Accuracy</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">F1-Score</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Precision</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Recall</th></tr></thead><tbody><tr><td align="left" valign="middle">HER2SC</td><td align="center" valign="middle">83.3%</td><td align="center" valign="middle">86.7%</td><td align="center" valign="middle">89.6%</td><td align="center" valign="middle">87.5%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">BRCA</td><td align="center" valign="middle" style="border-bottom:solid thin">53.8%</td><td align="center" valign="middle" style="border-bottom:solid thin">21.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">81.2%</td><td align="center" valign="middle" style="border-bottom:solid thin">31.5%</td></tr></tbody></table> </ephtml> </p> <p>Table 3 Results on the HER2SC test set.</p> <p> <ephtml> <table><thead><tr><th align="left" style="border-bottom:solid thin;border-top:solid thin">Method</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Accuracy</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">F1-Score</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Precision</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Recall</th></tr></thead><tbody><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>MLP Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">proposed method</td><td align="center" valign="middle" style="border-bottom:solid thin">83.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">86.7%</td><td align="center" valign="middle" style="border-bottom:solid thin">89.6%</td><td align="center" valign="middle" style="border-bottom:solid thin">87.5%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">48.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">39.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>Median Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td><td align="center" valign="middle" style="border-bottom:solid thin">43.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">78.6%</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">48.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">39.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>Mean Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td><td align="center" valign="middle" style="border-bottom:solid thin">43.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">78.6%</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">48.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">39.1%</td><td align="center" valign="middle" style="border-bottom:solid thin">62.5%</td></tr></tbody></table> </ephtml> </p> <p>Table 4 Results on the BRCA test set.</p> <p> <ephtml> <table><thead><tr><th align="left" style="border-bottom:solid thin;border-top:solid thin">Method</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Accuracy</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">F1-Score</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Precision</th><th align="center" style="border-bottom:solid thin;border-top:solid thin">Recall</th></tr></thead><tbody><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>MLP Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">proposed method</td><td align="center" valign="middle" style="border-bottom:solid thin">53.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">21.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">81.2%</td><td align="center" valign="middle" style="border-bottom:solid thin">31.5%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td><td align="center" valign="middle" style="border-bottom:solid thin">60.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">51.8%</td><td align="center" valign="middle" style="border-bottom:solid thin">72%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>Median Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td><td align="center" valign="middle" style="border-bottom:solid thin">12.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">7.8%</td><td align="center" valign="middle" style="border-bottom:solid thin">28.0%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">52.2%</td><td align="center" valign="middle" style="border-bottom:solid thin">63.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">66.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">72.0%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin"><bold>Mean Aggregation:</bold></td><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /><td align="left" valign="middle" style="border-bottom:solid thin" /></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">50.0%</td><td align="center" valign="middle" style="border-bottom:solid thin">12.3%</td><td align="center" valign="middle" style="border-bottom:solid thin">7.8%</td><td align="center" valign="middle" style="border-bottom:solid thin">28.0%</td></tr><tr><td align="left" valign="middle" style="border-bottom:solid thin">w/out pretrained CNN weights</td><td align="center" valign="middle" style="border-bottom:solid thin">52.2%</td><td align="center" valign="middle" style="border-bottom:solid thin">63.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">66.5%</td><td align="center" valign="middle" style="border-bottom:solid thin">72.0%</td></tr></tbody></table> </ephtml> </p> <hd id="AN0144773759-20">Author Contributions</hd> <p>Conceptualisation, S.P.O., J.R.P., T.G., J.S.C. and H.P.O.; Methodology, S.P.O., J.R.P., T.G. and J.S.C.; Data Curation, S.P.O. and R.C.-M.; Writing, S.P.O., J.R.P., T.G. and R.C.-M.; Review, J.S.C. and H.P.O.; Clinical Supervision, R.C.-M. and M.-J.C.; Technical Supervision, J.S.C. and H.P.O. All authors have read and agreed to the published version of the manuscript.</p> <hd id="AN0144773759-21">Funding</hd> <p>This work was partially funded by the Project "CLARE: Computer-Aided Cervical Cancer Screening" (POCI-01-0145-FEDER-028857), financially supported by FEDER through Operational Competitiveness Program–COMPETE 2020 and by National Funds through the Portuguese funding agency, FCT–Fundação para a Ciência e a Tecnologia, and also the FCT PhD grants "SFRH/BD/139108/2018" and "SFRH/BD/137720/2018".</p> <hd id="AN0144773759-22">Conflicts of Interest</hd> <p>The authors declare no conflict of interest.</p> <hd id="AN0144773759-23">Abbreviations</hd> <p>The following abbreviations are used in this manuscript:</p> <p></p> <p> <ephtml> <table><tbody><tr><td align="left" valign="middle">Adam</td><td align="left" valign="middle">Adaptive Moment Estimation</td></tr><tr><td align="left" valign="middle">ASCO/CAP</td><td align="left" valign="middle">American Society of Clinical Oncology/College of American Pathologists</td></tr><tr><td align="left" valign="middle">BCa</td><td align="left" valign="middle">Breast cancer</td></tr><tr><td align="left" valign="middle">BRCA</td><td align="left" valign="middle">TCGA-TCIA-BRCA</td></tr><tr><td align="left" valign="middle">CNN</td><td align="left" valign="middle">Convolutional Neural Network</td></tr><tr><td align="left" valign="middle">H&E</td><td align="left" valign="middle">Haematoxylin and Eosin</td></tr><tr><td align="left" valign="middle">HER2</td><td align="left" valign="middle">Human Epidermal growth factor Receptor 2</td></tr><tr><td align="left" valign="middle">HER2SC</td><td align="left" valign="middle">HER2 Scoring Contest dataset</td></tr><tr><td align="left" valign="middle">IHC</td><td align="left" valign="middle">Immunohistochemistry</td></tr><tr><td align="left" valign="middle">ISH</td><td align="left" valign="middle">In situ Hybridization</td></tr><tr><td align="left" valign="middle">MIL</td><td align="left" valign="middle">Multiple Instance Learning</td></tr><tr><td align="left" valign="middle">MLP</td><td align="left" valign="middle">Multilayer Perceptron</td></tr><tr><td align="left" valign="middle">ROI</td><td align="left" valign="middle">Regions of Interest</td></tr><tr><td align="left" valign="middle">WSI</td><td align="left" valign="middle">Whole Slide Images</td></tr></tbody></table> </ephtml> </p> <hd id="AN0144773759-24">Acknowledgments</hd> <p>The results published here are in whole or part based upon data generated by the TCGA Research Network: https://cancergenome.nih.gov/ and the HER2 Scoring Contest: https://warwick.ac.uk/fac/sci/dcs/research/tia/her2contest.</p> <ref id="AN0144773759-25"> <title> References </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> American Cancer Society Breast Cancer Facts & Figures 2017–2018Available online: https://<ulink href="http://www.cancer.org/content/dam/cancer-org/research/cancer-facts-and-statistics/breast-cancer-facts-and-figures/breast-cancer-facts-and-figures-2017-2018.pdf(accessed">www.cancer.org/content/dam/cancer-org/research/cancer-facts-and-statistics/breast-cancer-facts-and-figures/breast-cancer-facts-and-figures-2017-2018.pdf(accessed</ulink> on 21 June 2020)</bibtext> </blist> <blist> <bibl id="bib2" type="bt">2</bibl> <bibtext> Gandomkar Z., Brennan P., Mello-Thoms C. Computer-based image analysis in breast pathology. J. Pathol. Inform. 2016; 7. 10.4103/2153-3539.192814. 28066683</bibtext> </blist> <blist> <bibl id="bib3" idref="ref2" type="bt">3</bibl> <bibtext> Veta M., Pluim J.P.W., van Diest P.J., Viergever M.A. Breast Cancer Histopathology Image Analysis: A Review. IEEE Trans. Biomed. Eng. 2014; 61: 1400-1411. 10.1109/TBME.2014.2303852. 24759275</bibtext> </blist> <blist> <bibl id="bib4" idref="ref3" type="bt">4</bibl> <bibtext> American Society of Clinical Oncology (ASCO) Breast Cancer Guide. 2005–2020Available online: https://<ulink href="http://www.cancer.net/cancer-types/breast-cancer/introduction(accessed">www.cancer.net/cancer-types/breast-cancer/introduction(accessed</ulink> on 21 June 2020)</bibtext> </blist> <blist> <bibl id="bib5" idref="ref5" type="bt">5</bibl> <bibtext> Rakha E.A., Pinder S.E., Bartlett J.M.S., Ibrahim M., Starczynski J., Carder P.J., Provenzano E., Hanby A., Hales S., Lee A.H.S. Updated UK Recommendations for HER2 assessment in breast cancer. J. Clin. Pathol. 2015; 68: 93-99. 10.1136/jclinpath-2014-202571. 25488926</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Goddard K., Weinmann S., Richert-Boe K., Chen C., Bulkley J., Wax C. HER2 Evaluation and Its Impact on Breast Cancer Treatment Decisions. Public Health Genom. 2011; 15: 1-10. 10.1159/000325746</bibtext> </blist> <blist> <bibl id="bib7" idref="ref7" type="bt">7</bibl> <bibtext> Wolff A.C., Hammond M.E.H., Allison K.H., Harvey B.E., Mangu P.B., Bartlett J.M., Bilous M., Ellis I.O., Fitzgibbons P., Hanna W. Human Epidermal Growth Factor Receptor 2 Testing in Breast Cancer: American Society of Clinical Oncology/College of American Pathologists Clinical Practice Guideline Focused Update. J. Clin. Oncol. 2018; 36: 2105-2122. 10.1200/JCO.2018.77.8738</bibtext> </blist> <blist> <bibl id="bib8" idref="ref41" type="bt">8</bibl> <bibtext> Hanna W.M., Barnes P.J., Chang M., Gilks C.B., Magliocco A.M., Rees H., Quenneville L., Robertson S.J., Sengupta S.K., Nofech-Mozes S. Human epidermal growth factor receptor 2 testing in primary breast cancer in the era of standardized testing: A Canadian prospective study. J. Clin. Oncol. 2014; 32: 3967-3973. 10.1200/JCO.2014.55.6092. 25385731</bibtext> </blist> <blist> <bibl id="bib9" idref="ref9" type="bt">9</bibl> <bibtext> HEROHE ChallengeAvailable online: https://ecdp2020.grand-challenge.org/(accessed on 21 June 2020)</bibtext> </blist> <blist> <bibtext> Qaiser T., Mukherjee A., Reddy PB C., Munugoti S.D., Tallam V., Pitkäaho T., Lehtimäki T., Naughton T., Berseth M., Pedraza A. HER2 challenge contest: A detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues. Histopathology. 2018; 72: 227-238. 10.1111/his.13333</bibtext> </blist> <blist> <bibtext> Clark K., Vendt B., Smith K., Freymann J., Kirby J., Koppel P., Moore S., Phillips S., Maffitt D., Pringle M. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. J. Digit. Imaging. 2013; 26: 1045-1057. 10.1007/s10278-013-9622-7</bibtext> </blist> <blist> <bibtext> Lingle W., Erickson B.J., Zuley M.L., Jarosz R., Bonaccio E., Filippini J., Net J.M., Levi L., Morris E.A., Figler G.G. Radiology Data from The Cancer Genome Atlas Breast Invasive Carcinoma [TCGA-BRCA] collection. Cancer Imaging Arch. 2016. 10.7937/K9/TCIA.2016.AB2NAZRP</bibtext> </blist> <blist> <bibtext> GitHub Repository with CodeAvailable online: https://github.com/spoliveira/HERclassHE.git(accessed on 21 June 2020)</bibtext> </blist> <blist> <bibtext> Madabhushi A., Lee G. Image analysis and machine learning in digital pathology: Challenges and opportunities. Med. Image Anal. 2016; 33: 170-175. 10.1016/j.media.2016.06.037</bibtext> </blist> <blist> <bibtext> Robertson S., Azizpour H., Smith K., Hartman J. Digital image analysis in breast pathology—From image processing techniques to artificial intelligence. Transl. Res. 2017; 194: 19-35. 10.1016/j.trsl.2017.10.010. 29175265</bibtext> </blist> <blist> <bibtext> Tellez D., Balkenhol M., Otte-Höller I., van de Loo R., Vogels R., Bult P., Wauters C., Vreuls W., Mol S., Karssemeijer N. Whole-Slide Mitosis Detection in H E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks. IEEE Trans. Med Imaging. 2018; 37: 2126-2136. 10.1109/TMI.2018.2820199</bibtext> </blist> <blist> <bibtext> Cai D., Sun X., Zhou N., Han X., Yao J. Efficient Mitosis Detection in Breast Cancer Histology Images by RCNN. Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). Venice, Italy. 8–11 April 2019: 919-922. 10.1109/ISBI.2019.8759461</bibtext> </blist> <blist> <bibtext> Cruz-Roa A., Gilmore H., Basavanhally A., Feldman M., Ganesan S., Shih N., Tomaszewski J., Madabhushi A., González F. High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: Application to invasive breast cancer detection. PLoS ONE. 2018; 13e0196828. 10.1371/journal.pone.0196828. 29795581</bibtext> </blist> <blist> <bibtext> Li X., Radulovic M., Kanjer K., Plataniotis K.N. Discriminative Pattern Mining for Breast Cancer Histopathology Image Classification via Fully Convolutional Autoencoder. arXiv. 2019. 1902.08670. 10.1109/ACCESS.2019.2904245</bibtext> </blist> <blist> <bibtext> Romero F.P., Tang A., Kadoury S. Multi-Level Batch Normalization In Deep Networks For Invasive Ductal Carcinoma Cell Discrimination In Histopathology Images. arXiv. 2019. 1901.03684</bibtext> </blist> <blist> <bibtext> Wan T., Cao J., Chen J., Qin Z. Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features. Neurocomputing. 2017; 229: 34-44. 10.1016/j.neucom.2016.05.084</bibtext> </blist> <blist> <bibtext> Cao H., Bernard S., Heutte L., Sabourin R. Improve the Performance of Transfer Learning Without Fine-Tuning Using Dissimilarity-Based Multi-view Learning for Breast Cancer Histology Images. Image Analysis and RecognitionCampilho A., Karray F., Ter Haar Romeny B. ; Springer International Publishing: Berlin/Heidelberg, Germany. 2018: 779-787. 10.1007/978-3-319-93000-8_88</bibtext> </blist> <blist> <bibtext> Vo D.M., Nguyen N.Q., Lee S.W. Classification of breast cancer histology images using incremental boosting convolution networks. Inf. Sci. 2019; 482: 123-138. 10.1016/j.ins.2018.12.089</bibtext> </blist> <blist> <bibtext> Oscanoa J., Doimi F., Dyer R., Araujo J., Pinto J., Castaneda B. Automated segmentation and classification of cell nuclei in immunohistochemical breast cancer images with estrogen receptor marker. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando, FL, USA. 16–20 August 2016: 2399-2402. 10.1109/EMBC.2016.7591213</bibtext> </blist> <blist> <bibtext> Saha M., Chakraborty C., Arun I., Ahmed R., Chatterjee S. An Advanced Deep Learning Approach for Ki-67 Stained Hotspot Detection and Proliferation Rate Scoring for Prognostic Evaluation of Breast Cancer. Sci. Rep. 2017; 7. 10.1038/s41598-017-03405-5. 28607456</bibtext> </blist> <blist> <bibtext> Jamaluddin M.F., Fauzi M.F.A., Abas F.S., Lee J.T.H., Khor S.Y., Teoh K.H., Looi L.M. Cell Classification in ER-Stained Whole Slide Breast Cancer Images Using Convolutional Neural Network. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu, HI, USA. 18–21 July 2018: 632-635. 10.1109/EMBC.2018.8512386</bibtext> </blist> <blist> <bibtext> Rodner E., Simon M., Denzler J. Deep bilinear features for Her2 scoring in digital pathology. Curr. Dir. Biomed. Eng. 2017; 3: 811-814. 10.1515/cdbme-2017-0171</bibtext> </blist> <blist> <bibtext> Mukundan R. A Robust Algorithm for Automated HER2 Scoring in Breast Cancer Histology Slides Using Characteristic Curves. Medical Image Understanding and Analysis (MIUA); Springer: Cham, Switzerland. 2017; Volume 723: 386-397. 10.1007/978-3-319-60964-5_34</bibtext> </blist> <blist> <bibtext> Dou Q., Ouyang C., Chen C., Chen H., Heng P.A. Unsupervised Cross-Modality Domain Adaptation of Convnets for Biomedical Image Segmentations with Adversarial Loss. Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI); AAAI Press: Menlo Park, CA, USA. 2018: 691-697. 10.24963/ijcai.2018/96</bibtext> </blist> <blist> <bibtext> Aderghal K., Khvostikov A., Krylov A., Benois-Pineau J., Afdel K., Catheline G. Classification of Alzheimer Disease on Imaging Modalities with Deep CNNs Using Cross-Modal Transfer Learning. Proceedings of the 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS). Karlstad, Sweden. 18–21 June 2018: 345-350. 10.1109/CBMS.2018.00067</bibtext> </blist> <blist> <bibtext> Hadad O., Bakalo R., Ben-Ari R., Hashoul S., Amit G. Classification of breast lesions using cross-modal deep learning. Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). Melbourne, Australia. 18–21 April 2017: 109-112. 10.1109/ISBI.2017.7950480</bibtext> </blist> <blist> <bibtext> Bulten W., Bándi P., Hoven J., van de Loo R., Lotz J., Weiss N., van der Laak J., van Ginneken B., Hulsbergen-van de Kaa C., Litjens G. Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Sci. Rep. 2019; 9. 10.1038/s41598-018-37257-4</bibtext> </blist> <blist> <bibtext> Cruz-Roa A., Gilmore H.L., Basavanhally A., Feldman M.D., Ganesan S., Shih N.C., Tomaszewski J.P., Gonzalez F.A., Madabhushi A. Accurate and reproducible invasive breast cancer detection in whole-slide images: A Deep Learning approach for quantifying tumor extent. Sci. Rep. 2017; 7. 10.1038/srep46450. 28418027</bibtext> </blist> <blist> <bibtext> Honari S., Molchanov P., Tyree S., Vincent P., Pal C., Kautz J. Improving landmark localization with semi-supervised learning. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA. 19–21 June 2018: 1546-1555. 10.1109/CVPR.2018.00167</bibtext> </blist> <blist> <bibtext> Chapelle O., Wu M. Gradient descent optimization of smoothed information retrieval metrics. Inf. Retr. 2010; 13: 216-235. 10.1007/s10791-009-9110-3</bibtext> </blist> <blist> <bibtext> Sedeen Viewer SoftwareAvailable online: https://pathcore.com/sedeen/(accessed on 21 June 2020)</bibtext> </blist> <blist> <bibtext> Kingma D.P., Ba J. Adam: A method for stochastic optimization. arXiv. 2014. 1412.6980</bibtext> </blist> </ref> <aug> <p>By Sara P. Oliveira; João Ribeiro Pinto; Tiago Gonçalves; Rita Canas-Marques; Maria-João Cardoso; Hélder P. Oliveira and Jaime S. Cardoso</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref10"></nolink> <nolink nlid="nl2" bibid="bib11" firstref="ref11"></nolink> <nolink nlid="nl3" bibid="bib13" firstref="ref12"></nolink> <nolink nlid="nl4" bibid="bib14" firstref="ref13"></nolink> <nolink nlid="nl5" bibid="bib15" firstref="ref14"></nolink> <nolink nlid="nl6" bibid="bib16" firstref="ref15"></nolink> <nolink nlid="nl7" bibid="bib18" firstref="ref16"></nolink> <nolink nlid="nl8" bibid="bib21" firstref="ref17"></nolink> <nolink nlid="nl9" bibid="bib22" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib24" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib25" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib26" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib27" firstref="ref22"></nolink> <nolink nlid="nl14" bibid="bib28" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib29" firstref="ref25"></nolink> <nolink nlid="nl16" bibid="bib30" firstref="ref26"></nolink> <nolink nlid="nl17" bibid="bib31" firstref="ref27"></nolink> <nolink nlid="nl18" bibid="bib32" firstref="ref28"></nolink> <nolink nlid="nl19" bibid="bib33" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib34" firstref="ref34"></nolink> <nolink nlid="nl21" bibid="bib36" firstref="ref38"></nolink> <nolink nlid="nl22" bibid="bib37" firstref="ref39"></nolink>
CustomLinks:
  – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsdoj&genre=article&issn=20763417&ISBN=&volume=10&issue=14&date=20200701&spage=4728&pages=4728-4728&title=Applied Sciences&atitle=Weakly-Supervised%20Classification%20of%20HER2%20Expression%20in%20Breast%20Cancer%20Haematoxylin%20and%20Eosin%20Stained%20Slides&aulast=Sara%20P.%20Oliveira&id=DOI:10.3390/app10144728
    Name: Full Text Finder (for New FTF UI) (s8985755)
    Category: fullText
    Text: Find It @ SCU Libraries
    MouseOverText: Find It @ SCU Libraries
  – Url: https://doaj.org/article/c92b78d70228495793cbff626dc0ee9e
    Name: EDS - DOAJ (s8985755)
    Category: fullText
    Text: View record from DOAJ
    MouseOverText: View record from DOAJ
Header DbId: edsdoj
DbLabel: Directory of Open Access Journals
An: edsdoj.92b78d70228495793cbff626dc0ee9e
RelevancyScore: 893
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 892.696166992188
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sara+P%2E+Oliveira%22">Sara P. Oliveira</searchLink><br /><searchLink fieldCode="AR" term="%22João+Ribeiro+Pinto%22">João Ribeiro Pinto</searchLink><br /><searchLink fieldCode="AR" term="%22Tiago+Gonçalves%22">Tiago Gonçalves</searchLink><br /><searchLink fieldCode="AR" term="%22Rita+Canas-Marques%22">Rita Canas-Marques</searchLink><br /><searchLink fieldCode="AR" term="%22Maria-João+Cardoso%22">Maria-João Cardoso</searchLink><br /><searchLink fieldCode="AR" term="%22Hélder+P%2E+Oliveira%22">Hélder P. Oliveira</searchLink><br /><searchLink fieldCode="AR" term="%22Jaime+S%2E+Cardoso%22">Jaime S. Cardoso</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: Applied Sciences, Vol 10, Iss 14, p 4728 (2020)
– Name: Publisher
  Label: Publisher Information
  Group: PubInfo
  Data: MDPI AG, 2020.
– Name: DatePubCY
  Label: Publication Year
  Group: Date
  Data: 2020
– Name: Subset
  Label: Collection
  Group: HoldingsInfo
  Data: LCC:Technology<br />LCC:Engineering (General). Civil engineering (General)<br />LCC:Biology (General)<br />LCC:Physics<br />LCC:Chemistry
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22weakly-supervised+learning%22">weakly-supervised learning</searchLink><br /><searchLink fieldCode="DE" term="%22HER2%22">HER2</searchLink><br /><searchLink fieldCode="DE" term="%22breast+cancer%22">breast cancer</searchLink><br /><searchLink fieldCode="DE" term="%22Technology%22">Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Engineering+%28General%29%2E+Civil+engineering+%28General%29%22">Engineering (General). Civil engineering (General)</searchLink><br /><searchLink fieldCode="DE" term="%22TA1-2040%22">TA1-2040</searchLink><br /><searchLink fieldCode="DE" term="%22Biology+%28General%29%22">Biology (General)</searchLink><br /><searchLink fieldCode="DE" term="%22QH301-705%2E5%22">QH301-705.5</searchLink><br /><searchLink fieldCode="DE" term="%22Physics%22">Physics</searchLink><br /><searchLink fieldCode="DE" term="%22QC1-999%22">QC1-999</searchLink><br /><searchLink fieldCode="DE" term="%22Chemistry%22">Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22QD1-999%22">QD1-999</searchLink>
– Name: Abstract
  Label: Description
  Group: Ab
  Data: Human epidermal growth factor receptor 2 (HER2) evaluation commonly requires immunohistochemistry (IHC) tests on breast cancer tissue, in addition to the standard haematoxylin and eosin (H&E) staining tests. Additional costs and time spent on further testing might be avoided if HER2 overexpression could be effectively inferred from H&E stained slides, as a preliminary indication of the IHC result. In this paper, we propose the first method that aims to achieve this goal. The proposed method is based on multiple instance learning (MIL), using a convolutional neural network (CNN) that separately processes H&E stained slide tiles and outputs an IHC label. This CNN is pretrained on IHC stained slide tiles but does not use these data during inference/testing. H&E tiles are extracted from invasive tumour areas segmented with the HASHI algorithm. The individual tile labels are then combined to obtain a single label for the whole slide. The network was trained on slides from the HER2 Scoring Contest dataset (HER2SC) and tested on two disjoint subsets of slides from the HER2SC database and the TCGA-TCIA-BRCA (BRCA) collection. The proposed method attained 83.3 % classification accuracy on the HER2SC test set and 53.8 % on the BRCA test set. Although further efforts should be devoted to achieving improved performance, the obtained results are promising, suggesting that it is possible to perform HER2 overexpression classification on H&E stained tissue slides.
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: article
– Name: Format
  Label: File Description
  Group: SrcInfo
  Data: electronic resource
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2076-3417
– Name: NoteTitleSource
  Label: Relation
  Group: SrcInfo
  Data: https://www.mdpi.com/2076-3417/10/14/4728; https://doaj.org/toc/2076-3417
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.3390/app10144728
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://doaj.org/article/c92b78d70228495793cbff626dc0ee9e" linkWindow="_blank">https://doaj.org/article/c92b78d70228495793cbff626dc0ee9e</link>
– Name: AN
  Label: Accession Number
  Group: ID
  Data: edsdoj.92b78d70228495793cbff626dc0ee9e
PLink https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsdoj&AN=edsdoj.92b78d70228495793cbff626dc0ee9e
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3390/app10144728
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 1
        StartPage: 4728
    Subjects:
      – SubjectFull: weakly-supervised learning
        Type: general
      – SubjectFull: HER2
        Type: general
      – SubjectFull: breast cancer
        Type: general
      – SubjectFull: Technology
        Type: general
      – SubjectFull: Engineering (General). Civil engineering (General)
        Type: general
      – SubjectFull: TA1-2040
        Type: general
      – SubjectFull: Biology (General)
        Type: general
      – SubjectFull: QH301-705.5
        Type: general
      – SubjectFull: Physics
        Type: general
      – SubjectFull: QC1-999
        Type: general
      – SubjectFull: Chemistry
        Type: general
      – SubjectFull: QD1-999
        Type: general
    Titles:
      – TitleFull: Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sara P. Oliveira
      – PersonEntity:
          Name:
            NameFull: João Ribeiro Pinto
      – PersonEntity:
          Name:
            NameFull: Tiago Gonçalves
      – PersonEntity:
          Name:
            NameFull: Rita Canas-Marques
      – PersonEntity:
          Name:
            NameFull: Maria-João Cardoso
      – PersonEntity:
          Name:
            NameFull: Hélder P. Oliveira
      – PersonEntity:
          Name:
            NameFull: Jaime S. Cardoso
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 20763417
          Numbering:
            – Type: volume
              Value: 10
            – Type: issue
              Value: 14
          Titles:
            – TitleFull: Applied Sciences
              Type: main
ResultId 1