REPORTING THE OUTCOME
CosmosID and NIST will process and analyze all the raw sequence data and present our findings at the NIST Pathogen Detection Workshop in Gaithersburg Maryland on August 15th 2017. Remember, the goal of this Challenge is to assess bias associated with the measurement process, not grade individuals or labs on their competency. As such, results will be presented in aggregate and/or blinded. However, in the spirit of competition, we will “crown the winners” at the NIST workshop for the best data. Participants will be contacted before the workshop and given the option as to whether they wish to be personally identified and recognized by their peers.
We recognize that there is no single metric that can be used to assess “correct” or the “best”. Therefore, we are designating several metrics that can be used to assess performance (namely sensitivity and specificity). Examples of performance metrics include, but may not be limited to:
1. Specificity – Were the correct organisms identified? At what taxonomic level?
2. Sensitivity – Relative abundance of the individual components
3. Background Contaminants – False positives
4. Efficiency – Sensitivity and specificity after normalizing for sequencing depth
Following the August 2017 workshop, we will make all the raw sequence data publicly available.