presented By


When using metagenomic methods to assess the content of a complex microbial sample, there are many steps in the measurement process where bias might be introduced; including, but not limited to: sample collection and storage, DNA extraction method, library construction and/or PCR, NGS platform, raw data filtering and data analysis/interpretation. 


Investigating and understanding the source and extent of measurement bias is often accomplished through the development and utilization of a shared reference material (“ground truth”) followed by repeated measurements of the material in multiple independent laboratories. 


Welcome to The NIST-ABRF-CosmosID Metagenomic MVP Challenge v1.0


We have a metagenomic reference material (“ground truth”) and you are the independent laboratories! This challenge is open to the community at-large, but we are limiting the number of participants to the first 80 who sign-up. 


This Challenge (v1.0) is intended to be the first of many challenges that will each address a different step of the measurement process.  The goal of Challenge v1.0 is to specifically identify and understand measurement bias associated with 1) shotgun library preparation techniques and 2) next-generation sequencing platforms.   We recognize that this only addresses a small aspect of the larger measurement process.  This is intentional and it is our intent to address other steps of the measurement process in future challenges.




Genomic DNA From <25 Microbial Isolates

The DNA Mixture is Both Quantitative and Qualitative.

We know the genomes and we know their relative abundance


Etc. Etc. Etc. Etc.



Each participating laboratory receives its DNA mixture from NIST free of charge. Participants perform a library prep and sequencing using their preferred library prep method and sequencing platform.


For this challenge, in order to make data cross-comparable, sequencing techniques must be SHOTGUN METAGENOMIC approaches (NOT 16s amplicon).


The DNA Mixture: To be Sequenced by Participants

Participants use their preferred choice of library prep and NGS technology

Output to submit: FASTQ Raw Data and Metadata:
Library Prep and Sequencing Parameters



CosmosID and NIST will process and analyze all the raw sequence data and present our findings at the NIST Pathogen Detection Workshop in Gaithersburg Maryland on August 15th 2017.  Remember, the goal of this Challenge is to assess bias associated with the measurement process, not grade individuals or labs on their competency.  As such, results will be presented in aggregate and/or blinded.  However, in the spirit of competition, we will “crown the winners” at the NIST workshop for the best data.  Participants will be contacted before the workshop and given the option as to whether they wish to be personally identified and recognized by their peers. 


We recognize that there is no single metric that can be used to assess “correct” or the “best”. Therefore, we are designating several metrics that can be used to assess performance (namely sensitivity and specificity).  Examples of performance metrics include, but may not be limited to:


1.  Specificity – Were the correct organisms identified?  At what taxonomic level?


2.  Sensitivity – Relative abundance of the individual components


3.  Background Contaminants – False positives


4.  Efficiency – Sensitivity and specificity after normalizing for sequencing depth


Following the August 2017 workshop, we will make all the raw sequence data publicly available. 

Aug 15th 2017

NIST and CosmosID presents the results and announces a winner at the NIST Pathogen Detection Workshop



Other Challenges are Currently Being Organized:


1.  The “Strains” and “Standards” Challenge(s) are being organized by Janssen Human Microbiome Institute by Keng Cher Soh and Dirk Gevers


2. The Microbiome Quality Control (MBQC) Challenge v2.0 being hosted by Curtis Huttenhower et. al.


3. NIST Metagenomics MVP Challenge v 2.0 – Stay Tuned


IMMSA – International Microbiome and Metagenomics Standards Alliance: Hosted by NIST in collaboration with Cornell, U of Vermont, and Baylor College of Medicine. Monthly webex to discuss microbiome and metagenomic challenges

Register for the NIST Pathogen Workshop on August 15, 2017




Name *
Mailing Address (to ship DNA to): *
Mailing Address (to ship DNA to):