PrecisionFDA
Hidden Treasures - Warm Up

At the start of this challenge, users were provided with a NA12878 whole exome sequencing FASTQ file that had been in silico modified with single nucleotide variants (SNVs) and insertion and deletion (InDel) variants (less than 40 bp) at approximate Variant Allele Frequencies (VAFs) ≥20%. This precisionFDA note summarizes certain precisionFDA artifacts related to the challenge, including a VCF with the injected variants, as well as the VCF evaluation app which was used by the system to automatically evaluate all entries. We invite the community to try the evaluation app and give us any feedback.

Challenge participants ran the modified FASTQ file through their pipeline(s), and returned the resulting VCF file to precisionFDA for comparison to the known reference truth using a VCF evaluation app that we created. The first table below shows calls for 50 injected variants, including SNVs and InDels sorted separately by size (1-3 bp, 4-6 bp, 7-9 bp, 10-12 bp, and larger than 13 bp), for each entry in a separate row. Results shown for each entry also include whole exome recall, precision and F-score* for overall SNVs and overall InDels:

• Recall, or sensitivity, reflects the percentage of variants in the NIST/GiaB benchmark set that were exactly called by the challenge participant pipeline in a submitted dataset.
• Precision, or positive predictive value, reflects the percentage of called variants which match exactly the NIST/GiaB benchmark set.
• F-score is the harmonic mean of recall and precision, and is sometimes used as a single combined metric for evaluating overall accuracy.

The entries are sorted alphabetically by the name of the submitter, since many submitters had multiple entries. Moreover, the Entry column contains a link to the precisionFDA job that resulted from the execution of the evaluation app on that entry. All jobs, and the resulting files as output by the evaluation app, have been made public on precisionFDA and are owned by the special user 'Challenge Robot'.

Participants whose submissions correctly detected and called all 50 injected variants (regardless of the whole exome precision, recall and F-score for their pipeline/submission) are being recognized as Warm Up in silico Variant Catchers and are designated with the Variant Catcher icon in the table (45 entries earned that title).

The second table provides detailed variant detection information for each of the 50 injected variants from all qualified submissions. This table also specifies all injected variants with their expected VAF. In this table, VAF is displayed if the variant is present in submitted VCF for a specific entry and the VAF information was derived from the VCF file (i.e. VAF, AD/DP, or AB meta tag was available in the VCF file). For variants which were present in the VCF file, but failed to pass the filtering criteria (as defined by the filter meta tag), the variant is annotated as “filtered” with its associated VAF. Participant entries that supplied VAF data in accordance with the challenge instructions, and that detected all 50 variants reasonably close to the expected VAF values, are being recognized as Warm Up in silico VAF Spotters, indicated by the VAF Spotter icon in the table below (37 entries earned this title). These recognitions are based on the information found in raw VCF data for each entry, regardless of whether the submitter called each of the variants as above 20%. This is due to various reasons, including quality filters set for the pipeline.