Abstract

Transform the diagnostic delivery process in Mexican public hospitals by identifying and understanding the different stages of the process, with the goal of deploying intelligent traffic light signaling to support the diagnosis of breast cancer using convolutional neural networks and mammograms in a real-world environment.

Code

https://github.com/sanchezcarlosjr/breast-cancer-pipeline

Spanish slides

Introduction

Since breast cancer is the first dead cause in Mexico among women it became a big public health problem —in fact, 2.26 million cases worldwide. In other words, is a type of cancer with the highest incidence and mortality in women: every day at least 14 women, chiefly between 50 to 69 years, die. Indeed, as we can see in the below figure, breast cancer is an increasing tendency compared to other cancers.

Nowadays, doctors carried out analyses using Traditional 2D mammograms from patient requests at public Mexican hospitals. In Ensenada, doctors request private external assistance. Oncologists annotate medical images, and they chiefly say what is the patient's BI-RADS score. "BI-RADS" means Breast Imaging Reporting and Database System, and it's scoring standard radiologists and oncologists use to describe mammogram results. We'll explain further BI-RADS in the section.

Our goals are to build data mining models that understand mammograms and predict breast cancer developing risk, continuing works. Since our model output is a person's future healthy situation, we'll do descriptive and predictive methods, indeed we're going to apply Machine Learning algorithms to datasets. Of course, we don't expect to replace medical doctors but assist them. We know other computer-aided detection systems have been developed for breast cancer detection but no one applies them to regional cities and they are not free.

We expect our project can help thousands of women in quick cancer detection because deep learning is faster and cheaper than humans if we get good metrics, therefore we're contributing to the decrease in the death rate. PACS means Picture archiving and communication system.

Related work

The body of work related to breast cancer diagnostics using mammograms and convolutional neural networks is expansive. Various resources provide complementary perspectives, techniques, and tools.

For instance, the Open Health Imaging Foundation (OHIF) provides an open-source DICOM Viewer available on GitHub. The viewer is a zero-footprint medical image viewer provided as a Meteor package (OHIF, n.d.). It enables practitioners to visualize and navigate medical imaging data directly, enhancing understanding and improving diagnosis accuracy.

The Radiological Society of North America (RSNA) has published numerous papers discussing the importance of certain mammographic findings and terminologies. In one of these papers, they delve into the BI-RADS terminology for mammography reports, explaining what medical residents need to know (RSNA, 2023). This work informs the interpretation and communication of mammography results, which is a crucial step in diagnosing breast cancer.

Methodology

The first step in understanding the methodology for machine learning is familiarizing oneself with the key concepts. Data science is essentially the process of extracting meaningful insights and patterns from large and destructured datasets. This process leverages various techniques such as machine learning, neural networks, and statistical methodologies to decipher raw data, which can often be vast and complex.

An important tool in this context is the Digital Imaging and Communications in Medicine, or DICOM. DICOM is a standard protocol used for the transmission, storage, retrieval, and sharing of medical images. This protocol aids in the visualization and analysis of these images, enabling the identification of potential patterns or traits that might be of particular interest. In the world of data mining, this step is often referred to as "DICOM View."

In the context of mammography analysis, one might utilize the Digital Database for Screening Mammography, or DDSM. DDSM is one of the largest publicly available collections of mammograms. As part of the preprocessing step, the mammograms from DDSM can be analyzed and preprocessed to identify and potentially remove any noise or inconsistencies in the data. This process might involve data cleaning, normalization, transformation, and other techniques to prepare the data for further analysis.

Finally, the methodology wraps up with the engineering process. This is where you design and implement your deep learning or machine learning models based on the preprocessed data. This can involve creating different deep learning architechtures and preprocessing tasks. After training the model, you can then validate and test it on a separate dataset to ensure its reliability and effectiveness.

In this process, each step feeds into the next, creating a continuous flow from initial data understanding through to final model creation and evaluation. This framework allows for efficient handling and processing of complex and large-scale medical image data such as mammograms.

Domain Understanding

Breast cancer is a disease characterized by the abnormal and uncontrolled proliferation of cells, often leading to metastasis. It is commonly classified using the TNM staging system, which considers the size and extent of the tumor (T), the involvement of lymph nodes (N), and the presence of metastasis (M). The stage of the cancer is inversely related to survival rates; higher stages generally indicate a shorter lifespan, while lower stages are associated with longer survival.

Early diagnosis is crucial for improving outcomes, and various methods such as regular mammograms, self-exams, and awareness of risk factors are employed for this purpose. Speaking of risk factors, they can range from lifestyle choices like diet and obesity to chronic conditions, as well as environmental, familial, hereditary, benign, hormonal, and reproductive factors.

Additionally, the HER2 gene (Human Epidermal Growth Factor Receptor 2) can play a significant role in the development of breast cancer. Treatments targeting HER2 have been developed and are particularly effective for cases of HER2-positive breast cancer.

The mammography is the best technique to capture mammary microcalcifications.

Screening and basal

Mujeres asintomaticas con tumores en estadio precoces.

https://medicina.uc.cl/publicacion/10646/

Mammography

Craneo-caudal + ublicua medio lateral típicamente no benignas.

La mamografia con magnificación + lateral y/o focalización y/o tangencial. El objetivo es evaluar las lesiones pequenas, distorsiones y microcalcificaciones. El base a incidencias ortogonales, el radiologo indicara la zona a magnificar.

Amplacion radiologica.

Una calcificacion es una acumulación de calcio en el tejido corporal, haciendo que dicho tejido se endurezca (tiene capacidad para formar hueso).

Calficaciones mamarias, que evaluar tamano, localizacion, morfologia, distribucion. Los ultimos son descriptores BIRADS.

Las Ca+ asociadas al CDIS son en general microcalcificaciones: <0.5 mm.

Calcificaciones típicamente benignas

Vasculares

Groseras o en “pop corn”

Grandes, densas, >2-3 mm de diámetro

Fibroadenoma en involución

Lineales gruesas

Miden más de 1 mm, ectasia ductal, enf secretora o mastitis de células plasmáticas, intraductales, bilaterales, periductules.

Redondeadas o puntiformes

Anulares

Leche cálcica

Sedimento de Ca en macro-microquistes

Calcificaciones malignas

Risk factors

Factores de riesgo tales como dieta, estilo de vida (obesidad,

estado crónico), ambientales, familiares, hereditarios, condiciones benigma, hormonales y reproductivos.

HER2 (human epidermal growth factor receptor 2) is a gene that can play a role in the development of breast cancer.

Gail model.

BCRA1, BRCA2, PALB2, ATM, CHECK2

Carcinoma Lobullilar, Ductual in situ, hiperplasia atipica ductual, lobulillar.

Tamoxifeno 20 mg en premenospausia

Raloxifeno 60 mg en post menopausicas por 5 años

Inhibiedores de Aromatasa, exemestano, evidencia MAP-33, IBIS II6

Diagnostico temprano: tamizaje, autoexamen mensual a partir de los 18 años, aprox. día 10. Examen clínico anual a partir de los 25 años. Mastografía anual de tamizaje a partir de los 40 años. Consenso mexicano sobre diagnostico y tratamiento del cancer mamario. Decima reunion colima 2023.

La edad de corte (40 años), densidad de la mama debido a las limitaciones tecnologías.Tejido fibroboso.

Ultrasonido mamario a menos de 40 años con patología mamaria. Mastografía y ultra sonido son estudios complementarios, sensidibilidad 87%.

No, menos 25 años, pero 1 relacion directa 10 años.

Biopsia.

Norma Oficial Mexicana 041. American Cancer Society.

Tendencia/Riesgo

Conceptos claves de oncología. Un oncólogo es el médico que trata el cáncer y brinda atención médica para una persona diagnosticada con cáncer.

El experto en evaluar mastografías y otras imagenes son Radiologos. BIRADS es un lenguaje comun entre el radiologo y el oncologo. Nota: BIRADS es una historia antes del cancer. El cancer es una «siembra». Calcificaciones, morfología,…. Mastografía digital, ultrasonido mamario, resonancia magnética (casos especiales). Prevención y detección temprana.

La posibilidad de vivir es directamente proporcional con la Etapa clínica al DX, la posibilidades de Tto y la biología de la enfermedad.

Sobrevida global.

Sobrevida libre de progresión.

Benign tumors are noncancerous. Malignant tumors are cancerous.

ERBB3, HER2

https://www.cancer.org/es/cancer/tipos/cancer-de-seno/comprension-de-un-diagnostico-de-cancer-de-seno/estado-de-her2-del-cancer-de-seno.html

Fisica medica

Radioterapia

Acelerador de particulas

Cobalto

quimioterapia

Previous work

Datasets

DDSM

CBIS-DDSM

Big Data

Updating the ljpeg library was a crucial step in our workflow to download DDSM (Digital Database for Screening Mammography) images and other images in the LJPEG format for further algorithmic processing. The ljpeg library allows you to efficiently handle and decode images encoded in the LJPEG format.

By updating the library, we ensure that you have the latest version with any bug fixes or performance improvements.

https://github.com/sanchezcarlosjr/ljpeg

Exploratory data analysis

In our exploratory analysis of BIRADS cases (our label), we observe that the majority of cases fall into BIRADS 1 and 2, with 9371 and 15013 cases, respectively. BIRADS 3 and 4 have a relatively lower number of cases, with 1342 and 282 cases, respectively. BIRADS 5 has the lowest number of cases, with only 61 instances. BIRADS 0 represents a moderate number of cases, with 1806 instances.

Data labeling

We annotate data to define the ground truth, which is used to train the models. This process requires manual work. The task at hand dictates what must be annotated.

• Bounding boxes. This task aims to identify objects by rectangular boxes which locates the target object. We may represent those rectangular boxes by either $\{(x_1,y_1),(x_2,y_2)\}$ or $\{(x_1,y_1),weight,height\}$.

• Polygonal segmentation. This task aims to detect objects with polygons.

• Semantic segmentation. This task aims to classify every pixel in order to detect objects accurately.

• 3d cuboids. Similar to bounding boxes, they provide depth information.

• Key-Point and Landmarks. The task aims to locate key parts of objects in an image. Keypoints are the independent parameters that define the configuration of the mechanical system.

• Lines and Splines. Its purpose is detect curves in a image.

Formats

Because no standard format has been established, today we have various formats for annotating images, such as COCO, Pascal VOC, YOLO, TensorFlow TFRecord, and PyTorch txt.

Tools

We’ve segmented on Labelme, a graphical annotation tool written in Python that renders its graphical interface with Qt.

When we open the directory that holds the mammograms, we can work with those images and segment them manually.

Labelme automatically saves the segmented images in a JSON format, which makes it possible to reconstruct the segmented image with the command labelme_draw_json <PathToJSONFile>:

Following this procedure with other images:

Labeling pipeline

SAM

https://github.com/wkentaro/labelme/tree/main

https://github.com/HumanSignal/awesome-data-labeling

https://github.com/wkentaro/labelme

Pipeline experiments

In below experiments, various configurations and methods are explored to train a neural network. The experiments range from basic setups without data preprocessing to more advanced configurations involving various techniques like SAM, transformers, and different architectures like EfficientNet. Finally, the network that performs the best is further tested on a hospital dataset to evaluate its applicability in a real-world setting.

TwoViewDensityNet

We’ve reproduced the workflow proposed by [1] on a notebook.

Preprocessing images

Our preprocessing workflow is inspired by the approach used in the TwoViewDensityNet research. Their method, which delivers promising results on the Digital Database for Screening Mammography (DDSM), involves applying a binary mask, cropping the image, applying the magma colormap, and resizing the image to a dimension of 336x224.

We segment semantically with SAM to overlay mammae because it promises zero-shot learning, that is, a few annoted images are needed to get good results. The TIFF images are annotated with the COCO format and the PixelAnnotationTool. This approach can lead to a more detailed and comprehensive understanding of the mammography image by identifying and categorizing every single element in the image.

The next step is to apply morphological operations, a set of operations that process an image based on its shape. These operations can help eliminate noise, fill holes, or isolate individual elements.

Following that, we apply the mask to the image. The mask allows us to focus on the relevant part of the image and ignore the rest.

Then we apply the magma colormap again, which helps to visualize the image in a different color scheme that might bring out certain details better.

Finally, we resize the image, which can be necessary to standardize the input for our model, to ensure consistent results and to reduce computational demand.

These enhancements to the preprocessing stage aim to provide a more accurate representation of the data, improve the performance of subsequent analysis steps, and potentially lead to more reliable breast cancer detection results.

Including preprocessing with Ray for distributed processing was a significant addition to the workflow. By leveraging Ray, a distributed computing framework, we distribute the preprocessing tasks across multiple nodes or machines, enabling parallel processing and potentially reducing the overall processing time.

Architecture

Results

We got bad results even though we experiment with different loss functions as suggested in the paper.

Train a neural network without processed data and without the final layers.

Train TwoViewDensityNetwith new preprocessing steps, including SAM

Preprocessing

graph TD
  RawImage --> findCentroid
  findCentroid --> SAM
  RawImage --> Overlay
  SAM -->|mask| Overlay
  Overlay --> applyMagma
  applyMagma --> OutputImage

Fine tuning

Because raw SAM was not enough, we’re going to do fine tuning.

Use EfficientNet instead of ResNet50

We must switch from the Gradient Stochastic Descent optimizer to ADAM and the most modern solution EfficientNet.

Train the neural network with preprocessing, SAM, transformers, and four views

Test the best-performing network on a hospital dataset.

Metrics

IoU score

Accuracy

Deployment

Current Hospital Process

The current process for breast cancer detection in hospitals involves performing a mammogram, where the patient is fitted with a band for the study. Then, the image is sent to the the open-source application K-PACS, installed throughout the hospital. This system currently does not allow for quickly sharing studies with other hospitals or doctors, and there is a plan to implement a national-level electronic record called SINBA to centralize information.

Patients cannot consult the information in the data repository, which includes medical notes, and a unique population registration identity code (CURP) is required to access it. The images are automatically sent in a ZIP file to an external provider which evaluates them and delivers the results (BERAX). The hospitals pay this provider for the service.

There is a proposal to replace the sending of ZIP files with uncompressed images to speed up the process. Currently, the results can take between 1 and 6 days to be delivered, which can cause delays in the diagnosis and treatment of cancer. Patients need the results as quickly as possible so that they can undergo biopsies and additional tests.

We propose to implement a traffic light system for the status of the analyses, allowing users to know when the results are available. Additionally, work could be done on the implementation of an artificial intelligence (AI) project to improve the image analysis process, although this is not foreseen in the short or medium term.

The success of the AI project could attract funding and support for other similar projects in the future. To achieve this, it is important that specific responsibilities are identified and assigned to the people who will feed the necessary information into the system.

DICOM

DICOM stands for Digital Imaging and Communications in Medicine. It's a global standard for handling, storing, printing, and transmitting information in medical imaging. The standard was created by the National Electrical Manufacturers Association (NEMA) and is widely used in hospitals globally. It includes a file format and a network communications protocol, and it defines data structures for medical images and related information like patient data, image acquisition parameters, and diagnostic findings.

DICOM enables the integration of scanners, servers, workstations, printers, and network hardware from multiple manufacturers into a picture archiving and communication system (PACS). The different devices come with DICOM conformance statements that state how they support the DICOM standard.

DICOMWeb, on the other hand, is a term used to denote the family of DICOM RESTful web services. These web services are defined as part of the DICOM standard and provide access to a set of fundamental DICOM functions using familiar web technologies. They are an HTTP-based API for the DICOM protocol, making it more accessible to web-based applications.

There are several services defined under the DICOMWeb umbrella, but we’ve implemented QIDO (Query based on ID for DICOM Objects) with ConQuest DICOM server 1.5.0c and OHIF Viewer.

ConQuest DICOM server released in 1995 by Marcel Van Herk, is a widely used, versatile DICOM server. This server can be found on GitHub and the client https://github.com/sanchezcarlosjr/MexicanPACS.

This version of the server was used by the University of California at Davis for their Personal PACS (Picture Archiving and Communication System). PACS is essential in modern healthcare since it allows for the storage and convenient access of medical images. Personal PACS systems can provide substantial benefits to healthcare professionals by enabling access to patient images and related data from their personal devices.

The ConQuest DICOM server uses a Delphi TCP/IP connection for network communication. Delphi is a programming language and software development kit that supports Windows APIs, including those for establishing and managing TCP/IP connections.

The server software also includes Lua scripting. Lua is a lightweight and efficient scripting language commonly used for extending applications. It provides the capability to incorporate advanced logic into the server's operation without modifying the server's source code.

Overall, the ConQuest DICOM server is an essential tool for managing medical imaging data, particularly in research and clinical contexts where flexibility and customizability are paramount.

On the another hand, medical doctors who wish to analyze mammographies are using DICOM Viewer. In our case, we’ll customize OHIF Viewer.

Our model is going to be deployed over Gradio such that some background service can call our model as API REST.

Conclusions

References

Load and preprocess images  |  TensorFlow Core

[1] TwoViewDensityNet: Two-View Mammographic Breast Density Classification Based on Deep Convolutional Neural Network Mariam Busaleh 1 , Muhammad Hussain 1,* , Hatim A. Aboalsamh 1 , Fazal-e-Amin 2 and Sarah A. Al Sultan 3

Open Health Imaging Foundation. (n.d.). OHIF/Viewers. GitHub. Retrieved June 13, 2023, from https://github.com/OHIF/Viewers

Radiological Society of North America. (n.d.). Radiol.211105. RSNA Journals. Retrieved June 13, 2023, from https://pubs.rsna.org/doi/10.1148/radiol.211105

Sanchez Carlos Jr. (n.d.). Breast-Cancer-risk-estimation-system. GitHub. Retrieved June 13, 2023, from https://github.com/sanchezcarlosjr/Breast-Cancer-risk-estimation-system

Society for Imaging Informatics in Medicine. (n.d.). SIIM. Retrieved June 13, 2023, from https://siim.org/

Ray Project. (n.d.). Pipelining Datasets. Ray Documentation. Retrieved June 13, 2023, from https://docs.ray.io/en/latest/data/pipelining-compute.html#pipelining-datasets

Ray Project. (n.d.). ray.data.read_images. Ray Documentation. Retrieved June 13, 2023, from https://docs.ray.io/en/latest/data/api/doc/ray.data.read_images.html#ray.data.read_images

Ray Project. (n.d.). OCR Example. Ray Documentation. Retrieved June 13, 2023, from https://docs.ray.io/en/latest/data/examples/ocr_example.html

Radiological Society of North America. (2023, May 10). BI-RADS Terminology for Mammography Reports: What Residents Need to Know. RSNA Journals. Retrieved June 13, 2023, from https://pubs.rsna.org/do/10.1148/rg.2019180068.pres/full

Sanchez Carlos Jr. (n.d.). breast-cancer-pipeline. GitHub. Retrieved June 13, 2023, from https://github.com/sanchezcarlosjr/breast-cancer-pipeline

https://github.com/Adamouization/Breast-Cancer-Detection-Mammogram-Deep-Learning-Publication

https://www.thelancet.com/journals/landig/article/PIIS2589-7500(23)00153-X/fulltext

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0280841

https://github.com/Project-MONAI/MONAI

https://www.youtube.com/@residenciaimageneshigasanm7739

https://www.youtube.com/watch?v=-KGaoQX6OVQ

https://github.com/SysCV/sam-hq

https://www.imss.gob.mx/sites/all/statics/guiasclinicas/240GRR.pdf

Annexes

Preprocesamiento de mamografías para segmentar la mama

Management