CFSAN Pathogen Detection Challenge
Identify the types and distribution of Salmonella strains in metagenomics samples
2018-02-16 01:00:00 UTC
2018-04-26 23:59:59 UTC
Results - Metagenomics Challenge
The precisionFDA Pathogen Detection Challenge ran from February 15th, 2018 to April 26th, 2018. The challenge provided the participants with the opportunity to test their ability to detect Salmonella in shotgun metagenomic samples from contaminated cilantro. The goal of the challenge was to identify and type Salmonella in naturally and in silico contaminated samples. There were twenty-six entries from eleven different organizations that participated in the challenge.
Participants were provided with a set of gzipped FASTQ files containing eight training samples and twenty-four challenge samples. Training samples were spiked with either low or high levels of Salmonella Newport. Challenge samples had been previously characterized using traditional microbiological culturing techniques. Challenge samples were either culture positive (eight samples), culture negative (eleven samples), or culture negative samples that were in silico spiked with Salmonella reads (five samples). Participants were asked to identify the serotype, sequence type (i.e. Multi-Locus Sequence Typing), and strain of Salmonella present in positive challenge samples.
Overview of Results
The table below provides an overview of the submission results. Submissions were evaluated according to the number of culture negative samples correctly identified (negative_count), number of spiked in silico samples correctly identified as positive (synthetic_detected_count), number of spiked in silico samples with correct serotype (synthetic_serotype_count), number of spiked in silico samples with correct MLST (synthetic_mlst_count), number of spiked in silico samples with correct strain (synthetic_strain_count), number of culture positive samples correctly identified as positive (positive_detected_count), number of culture positive samples with correct serotype (positive_serotype_count), number of culture positive samples with correct MLST (positive_mlst_count), and number of culture positive samples with correct strain (positive_strain_count).
There were a total of 11 negative samples, 5 synthetic samples, and 8 positive samples.
The accuracy of submissions was evaluated at three taxonomic levels (serotype, MLST, and strain). Five submitters correctly identified serotypes in nine of the thirteen Salmonella positive samples (four in silico and five culture positive samples). Two submitters correctly identified MLST types in nine of the thirteen Salmonella positive samples (five in silico and four culture positive samples). One submitter correctly identified strains in eight of the thirteen Salmonella positive samples (five in silico and three culture positive samples).
The challenge data included eleven samples that were negative for Salmonella using traditional microbiological culturing techniques. While we would expect culturing to be more sensitive for detecting Salmonella at the sequencing depths used for these samples we cannot exclude the possibility that a culture negative sample contained Salmonella. For this reason we did not consider culture negative samples in the final scoring.
The full challenge key with annotations as to which samples were which is available here.