ISIC 2016 Challenge [Closed]

Summary

The overarching goal of the challenge is to develop image analysis tools to enable the automated diagnosis of melanoma from dermoscopic images. Image analysis of skin lesions is composed of 3 parts:

  • Part 1: Lesion Segmentation
  • Part 2: Detection and Localization of Visual Dermoscopic Features/Patterns
  • Part 3: Disease Classification

This challenge provides training data (~900 images) for participants to engage in all 3 components of lesion image analysis. A separate test dataset (~350 images) will be provided for participants to generate and submit automated results.

Background

Melanoma

Skin cancer is a major public health problem, with over 5 million newly diagnosed cases in the United States each year. Melanoma is the deadliest form of skin cancer, responsible for over 9,000 deaths each year.

Melanoma Tutorial

Dermoscopy

As pigmented lesions occurring on the surface of the skin, melanoma is amenable to early detection by expert visual inspection. It is also amenable to automated detection with image analysis. Given the widespread availability of high-resolution cameras, algorithms that can improve our ability to screen and detect troublesome lesions can be of great value. As a result, many centers have begun their own research efforts on automated analysis. However, a centralized, coordinated, and comparative effort across institutions has yet to be implemented.

Dermoscopy is an imaging technique that eliminates the surface reflection of skin. By removing surface reflection, visualization of deeper levels of skin is enhanced. Prior research has shown that when used by expert dermatologists, dermoscopy provides improved diagnostic accuracy, in comparison to standard photography. As inexpensive consumer dermatoscope attachments for smart phones are beginning to reach the market, the opportunity for automated dermoscopic assessment algorithms to positively influence patient care increases.

Dermoscopic Image Analysis

ISIC Archive

The International Skin Imaging Collaboration (ISIC) is an international effort to improve melanoma diagnosis, sponsored by the International Society for Digital Imaging of the Skin (ISDIS). The ISIC Archive contains the largest publicly available collection of quality controlled dermoscopic images of skin lesions.

Presently, the ISIC Archive contains over 13,000 dermoscopic images, which were collected from leading clinical centers internationally and acquired from a variety of devices within each center. Broad and international participation in image contribution is designed to insure a representative clinically relevant sample. Initial data sets derive primarily from the United States, and the ISIC has ongoing commitments and contributions of additional data sets from international collaborators.

All incoming images to the ISIC Archive are screened for both privacy and quality assurance. Most images have associated clinical metadata, which has been vetted by recognized melanoma experts. A subset of the images have undergone annotation and markup by recognized skin cancer experts. These markups include dermoscopic features (i.e., global and focal morphologic elements in the image known to discriminate between types of skin lesions).

The software infrastructure of the ISIC Archive is built using the open-source Girder platform, and the source code for the Archive itself is freely available on GitHub. All the data and annotations presently within the ISIC Archive are licensed for free public download and use, under the CC-0 license.

Challenge Details

Timeline (subject to change)

  • Jan 5: Release initial data (images + ground truth)
  • March 18: Release test data for Part 1 (images)
  • March 25: Release test data for Part 2-3
  • Apr. 1: Part 1, 2, 3 submission deadline
  • Apr. 8: Part 2B, 3B submission deadline
  • Apr. 14: Winners announced

Challenge Tasks

Part 1: Lesion Segmentation. Participants are asked to submit automated predictions of lesion segmentations from dermoscopic images. Human expert ground truth will be provided as the comparison standard. Performance metrics used will include the dice coefficient, as well as pixel level accuracy, specificity, sensitivity, and average precision at sensitivity of 100%.

Part 2: Lesion Dermoscopic Feature Extraction. Participants are challenged to submit automated predictions of dermoscopic features, including both localization and classification. Expert-annotated dermatological ground truth will be provided as the comparison standard. Performance metrics used will include the dice coefficient, as well as pixel level accuracy, specificity, and sensitivity. Participants will be ranked according to average precision. The chosen features are streaks (incl. starburst pattern) and globules.

Part 3: Lesion Classification. Participants are challenged to extract their own features in order to predict lesion disease state of benign or malignant. Performance metrics will include accuracy, specificity, sensitivity, specificity at 99% sensitivity, and average precision evaluated at sensitivity of 100%. Participants will be ranked according to average precision.

Datasets

The "Skin Lesion Analysis Towards Melanoma Detection" challenge leverages a dataset of annotated skin lesion images from the ISIC Archive, The dataset contains a representative mix of images of both malignant and benign skin lesions, Before release, the challenge dataset was randomly partitioned into both a training and test sets, with ~900 images in the training set and ~350 images in the test set.

Participants are allowed to use prior work and outside data sources, but works must be cited and participants will be placed into a different category for evaluation.

Reference Standard

There will be 3 reference standard annotations for the 3 sub-challenges.

  • Segmentation: Lesions have been segmented against background normal skin and miscellaneous structures by expert dermatologists.
  • Dermoscopic Features: Lesions have been locally annotated for clinical dermoscopic features by expert dermatologists.
  • Disease State: The gold standard for diagnosis of skin lesions is pathology. Images in the ISIC archive have been derived from centers with expert pathology that can be deemed the gold standard. Benign lesions included in the archive without benefit of pathology diagnosis are reviewed by multiple experts and only included in the event of unanimous clinical diagnosis.

Journal Paper & Data Citation

Updated: May 11th, 2016

An article has been posted to arXiv, where we have summarized the dataset, the evaluation protocol, and preliminary analyses of the results. We strongly encourage you to write manuscripts describing your approaches and submit for publication to peer reviewed journals, or to arXiv. As you publish your work, please use the following citation to refer to the challenge:

Gutman, David; Codella, Noel C. F.; Celebi, Emre; Helba, Brian; Marchetti, Michael; Mishra, Nabin; Halpern, Allan. "Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)". eprint arXiv:1605.01397. 2016.

This paper is accessible from the following link: https://arxiv.org/abs/1605.01397

Contact Information

Support for the challenge can be attained by contacting the sponsors at isbi-sponsors@isic-archive.com.

Organizers

  • The International Skin Imaging Collaboration
  • Allan Halpern (Clinical Leader) (Memorial Sloan Kettering Cancer Center, New York, USA)
  • Brian Helba (Technical Leader) (Kitware, New York, USA)
  • David Gutman (Technical Leader) (Emory University, Atlanta, USA)
  • Noel Codella (Image Analysis) (IBM, New York, USA)
  • M. Emre Celebi (Image Analysis) (Louisiana State University, Shreveport, Louisiana)