ISIC 2016 Challenge [Closed]

Summary

The overarching goal of the challenge is to develop image analysis tools to enable the automated diagnosis of melanoma from dermoscopic images. Image analysis of skin lesions is composed of 3 parts:

  • Part 1: Lesion Segmentation
  • Part 2: Detection and Localization of Visual Dermoscopic Features/Patterns
  • Part 3: Disease Classification

This challenge provides training data (~900 images) for participants to engage in all 3 components of lesion image analysis. A separate test dataset (~350 images) will be provided for participants to generate and submit automated results.

Challenge Details

Timeline (subject to change)

  • Jan 5: Release initial data (images + ground truth)
  • March 18: Release test data for Part 1 (images)
  • March 25: Release test data for Part 2-3
  • Apr. 1: Part 1, 2, 3 submission deadline
  • Apr. 8: Part 2B, 3B submission deadline
  • Apr. 14: Winners announced

Challenge Tasks

Part 1: Lesion Segmentation. Participants are asked to submit automated predictions of lesion segmentations from dermoscopic images. Human expert ground truth will be provided as the comparison standard. Performance metrics used will include the dice coefficient, as well as pixel level accuracy, specificity, sensitivity, and average precision at sensitivity of 100%.

Part 2: Lesion Dermoscopic Feature Extraction. Participants are challenged to submit automated predictions of dermoscopic features, including both localization and classification. Expert-annotated dermatological ground truth will be provided as the comparison standard. Performance metrics used will include the dice coefficient, as well as pixel level accuracy, specificity, and sensitivity. Participants will be ranked according to average precision. The chosen features are streaks (incl. starburst pattern) and globules.

Part 3: Lesion Classification. Participants are challenged to extract their own features in order to predict lesion disease state of benign or malignant. Performance metrics will include accuracy, specificity, sensitivity, specificity at 99% sensitivity, and average precision evaluated at sensitivity of 100%. Participants will be ranked according to average precision.

Datasets

The "Skin Lesion Analysis Towards Melanoma Detection" challenge leverages a dataset of annotated skin lesion images from the ISIC Archive, The dataset contains a representative mix of images of both malignant and benign skin lesions, Before release, the challenge dataset was randomly partitioned into both a training and test sets, with ~900 images in the training set and ~350 images in the test set.

Participants are allowed to use prior work and outside data sources, but works must be cited and participants will be placed into a different category for evaluation.

Reference Standard

There will be 3 reference standard annotations for the 3 sub-challenges.

  • Segmentation: Lesions have been segmented against background normal skin and miscellaneous structures by expert dermatologists.
  • Dermoscopic Features: Lesions have been locally annotated for clinical dermoscopic features by expert dermatologists.
  • Disease State: The gold standard for diagnosis of skin lesions is pathology. Images in the ISIC archive have been derived from centers with expert pathology that can be deemed the gold standard. Benign lesions included in the archive without benefit of pathology diagnosis are reviewed by multiple experts and only included in the event of unanimous clinical diagnosis.

Journal Paper & Data Citation

Updated: May 11th, 2016

An article has been posted to arXiv, where we have summarized the dataset, the evaluation protocol, and preliminary analyses of the results. We strongly encourage you to write manuscripts describing your approaches and submit for publication to peer reviewed journals, or to arXiv. As you publish your work, please use the following citation to refer to the challenge:

Gutman, David; Codella, Noel C. F.; Celebi, Emre; Helba, Brian; Marchetti, Michael; Mishra, Nabin; Halpern, Allan. "Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)". eprint arXiv:1605.01397. 2016.

This paper is accessible from the following link: https://arxiv.org/abs/1605.01397

Contact Information

Support for the challenge can be attained by contacting the sponsors at support@isic-archive.com.

Organizers

  • The International Skin Imaging Collaboration
  • Allan Halpern (Clinical Leader) (Memorial Sloan Kettering Cancer Center, New York, USA)
  • Brian Helba (Technical Leader) (Kitware, New York, USA)
  • David Gutman (Technical Leader) (Emory University, Atlanta, USA)
  • Noel Codella (Image Analysis) (IBM, New York, USA)
  • M. Emre Celebi (Image Analysis) (Louisiana State University, Shreveport, Louisiana)