PrecisionFDA
CFSAN Pathogen Detection Challenge


Identify the types and distribution of Salmonella strains in metagenomics samples


  • Starts
    2018-02-16 01:00:00 UTC
  • Ends
    2018-04-26 23:59:59 UTC

Results - Metagenomics Challenge

The precisionFDA Pathogen Detection Challenge ran from February 15th, 2018 to April 26th, 2018. The challenge provided the participants with the opportunity to test their ability to detect Salmonella in shotgun metagenomic samples from contaminated cilantro. The goal of the challenge was to identify and type Salmonella in naturally and in silico contaminated samples. There were twenty-six entries from eleven different organizations that participated in the challenge.

Introductory Remark

Participants were provided with a set of gzipped FASTQ files containing eight training samples and twenty-four challenge samples. Training samples were spiked with either low or high levels of Salmonella Newport. Challenge samples had been previously characterized using traditional microbiological culturing techniques. Challenge samples were either culture positive (eight samples), culture negative (eleven samples), or culture negative samples that were in silico spiked with Salmonella reads (five samples). Participants were asked to identify the serotype, sequence type (i.e. Multi-Locus Sequence Typing), and strain of Salmonella present in positive challenge samples.

Overview of Results

The table below provides an overview of the submission results. Submissions were evaluated according to the number of culture negative samples correctly identified (negative_count), number of spiked in silico samples correctly identified as positive (synthetic_detected_count), number of spiked in silico samples with correct serotype (synthetic_serotype_count), number of spiked in silico samples with correct MLST (synthetic_mlst_count), number of spiked in silico samples with correct strain (synthetic_strain_count), number of culture positive samples correctly identified as positive (positive_detected_count), number of culture positive samples with correct serotype (positive_serotype_count), number of culture positive samples with correct MLST (positive_mlst_count), and number of culture positive samples with correct strain (positive_strain_count).

There were a total of 11 negative samples, 5 synthetic samples, and 8 positive samples.

Submitter Organization Entry results negative_count synthetic_detected_count synthetic_serotype_count synthetic_mlst_count synthetic_strain_count positive_detected_count positive_serotype_count positive_mlst_count positive_strain_count
Mark Mammel FDA job-FG8gxgQ06B2yY1qp2GP90KxJ file-FG8gz380QXy9fK840k7k4x04 11 5 4 0 0 3 2 0 0
Thang Tran BioTuring job-FG8gxgj06B2yVB5p44XbyYFQ file-FG8gz680b1XXyVzj1p7228y3 11 2 0 0 0 7 0 0 0
Mark Mammel FDA job-FG8gxgj06B2qz55Q9898Vk2V file-FG8gz5Q0YX2zgFXF4p06ZXZp 11 5 3 2 3 4 4 2 2
Thang Tran BioTuring job-FG8gxgj06B2y9Pfj3xY446Z1 file-FG8gz700pFqpFJZ10kPZ5GYQ 11 4 0 1 0 7 0 1 0
Richa Agarwala NCBI job-FG8gxgj06B2Vv4Y45g88JpbP file-FG8gz6j0gpkVkzQG4Yy1YQZ1 11 5 5 2 3 5 5 1 3
Rainer Follador job-FG8gxj006B2Qz4kV41ff9Bjz file-FG8gz6Q005qKzfyq1zPq2v7j 3 4 2 2 0 7 4 1 1
Rongshan Yu Aginome Scientific job-FG8gxj006B2pZP2VKVVb3YFV file-FG8gz600V4Q10zY10jQQq30g 10 5 5 5 4 5 5 4 1
Richa Agarwala NCBI job-FG8gxj806B2jYyJ0083z0Z5G file-FG8gz6Q0JK68FzY30jbjvJ6B 10 5 5 2 3 6 5 1 3
Richa Agarwala NCBI job-FG8gxj806B2Vx6KbBfGBYx2F file-FG8gz900Qk060Q5V0jpgpPbz 10 5 5 2 3 6 5 1 3
Richa Agarwala NCBI job-FG8gxjQ06B2jVjP21BQzbGJZ file-FG8gz880G38Pv70kGBGByXyp 10 5 5 2 3 6 5 1 3
Bert Gold job-FG8gxjQ06B2xZ7Pg1BQG30BK file-FG8gzFj0p9jBvJGx0x77132J 8 0 0 0 0 7 2 0 0
Rongshan Yu Aginome Scientific job-FG8gxjj06B2Vv4Y45g88JpbV file-FG8gz9j044B7B8fy0k1GvVFp 10 5 5 5 4 5 5 4 2
Nur Hasan CosmosID job-FG8gxjj06B2ZVQV6882xX8bF file-FG8gz7006yjyGz08GBZgyB8F 11 5 5 1 5 5 5 3 3
Ryan Poplin Google job-FG8gxjj06B2fVZYp40933zKp file-FG8gz8Q0g1210zY10jQQq324 8 5 5 1 4 7 5 0 2
Adam Rivers USDA-ARS job-FG8gxk006B2xZ7Pg1BQG30BQ file-FG8gz7003gXBvJGx0x77131k 10 5 5 5 1 5 3 2 0
Altti Maarala job-FG8gxk806B2z53b2G5qb86Q7 file-FG8gz8Q0jyy7Pf4y4Z1F7qK6 11 4 1 2 0 3 2 2 0
Rongshan Yu Aginome Scientific job-FG8gxkQ06B2yY1qp2GP90KxY file-FG8gz700vGkgQ1Fp4P96315V 10 5 5 5 1 5 5 4 2
Nick Greenfield One Codex job-FG8gxkj06B2qGFvy5xf69gpv file-FG8gzBQ0Yqyzv70kGBGByXz3 11 5 4 5 4 5 5 4 3
Nick Greenfield One Codex job-FG8gxkj06B2Vx6KbBfGBYx2J file-FG8gzB008KyVkzQG4Yy1YQjq 10 5 4 5 4 6 5 4 3
Nick Greenfield One Codex job-FG8gxkj06B2yY1qp2GP90Kxb file-FG8gzBj0ppg2yVzj1p722904 8 5 4 5 4 6 5 4 3
Vesna Pajic Seven Bridges Genomics job-FG8gxp806B2gkxXVG5yXPqvb file-FG8gzYj0gK6BvJGx0x7713BK 9 5 3 2 4 5 1 0 0
Vesna Pajic Seven Bridges Genomics job-FG8gxpQ06B2gGxg21zbjX3gQ file-FG8gzg80bgxxfg3q4XZyPkqf 9 5 4 4 4 5 0 0 0
Vesna Pajic Seven Bridges Genomics job-FG8gxpj06B2QGgyJPBk7gQ9b file-FG8gzgj084B91XgX2Qf3kFVV 2 5 5 3 5 8 3 0 2
Nur Hasan CosmosID job-FG8gxpj06B2y5y9j5Pgb2yjk file-FG8gzj00p85zv70kGBGByYFj 11 5 5 1 5 4 4 3 3
Nemanja Vucic Seven Bridges Genomics job-FG8gxq006B2Vx6KbBfGBYx2P file-FG8gzk00Jxxq35bq0kk1qK73 10 5 5 3 5 5 3 0 2
Bert Gold job-FG8gxq006B2bVbBF8pZVxbg7 file-FG8gzkQ047p87gp1GBP9p3pG 11 0 0 0 0 5 3 0 0

The accuracy of submissions was evaluated at three taxonomic levels (serotype, MLST, and strain). Five submitters correctly identified serotypes in nine of the thirteen Salmonella positive samples (four in silico and five culture positive samples). Two submitters correctly identified MLST types in nine of the thirteen Salmonella positive samples (five in silico and four culture positive samples). One submitter correctly identified strains in eight of the thirteen Salmonella positive samples (five in silico and three culture positive samples).

The challenge data included eleven samples that were negative for Salmonella using traditional microbiological culturing techniques. While we would expect culturing to be more sensitive for detecting Salmonella at the sequencing depths used for these samples we cannot exclude the possibility that a culture negative sample contained Salmonella. For this reason we did not consider culture negative samples in the final scoring.

The full challenge key with annotations as to which samples were which is available here.