NDACC Protocol for the Validation of Instruments and Data Analysis Methods
Introduction
The Network for the Detection of Atmospheric Composition Change (NDACC) Validation Protocol is designed to insure that archived NDACC data are of as high a quality as possible, within the constraints of measurement technology and retrieval theory at the time the data were taken and analyzed. This is essential for the early detection of changes in atmospheric composition with the accuracy and precision necessary to understand the responsible forcings.
Validation is a continuing process. Instruments and their associated data analysis methods must be evaluated and verified before they are accepted in the NDACC, and must be continuously monitored throughout their use. This document serves as an outline plan for the validation process and builds upon the requirements detailed in the NDACC Measurements Protocol. The specific methods whereby individual instruments should be validated have been established by the various NDACC Instrument Working Groups (IWGs) and are outlined in the instrument-specific Validation Appendices that are provided along with this document. In addition, the NDACC Instrument Intercomparisons Protocol should be referenced for details regarding recommended intercomparison procedures.
The requirements for NDACC data providers have been stated in the NDACC Data Protocol. Information on areas in which modeling expertise can assist in the production of high quality data from various instruments can be found in the NDACC Theory and Analysis Protocol. Both of these protocols should be referred to in conjunction with this document.
Evaluation of New Instruments and Instrument Teams
The evaluation procedure for the acceptance of an instrument proposed for NDACC affiliation is detailed in the NDACC Measurements Protocol. The appropriate IWGs base their recommendation for acceptance on the following criteria.
Quality Criteria
These criteria are detailed for specific instrument types in the Validation Appendices and address issues such as:
- Evaluation of the instrument design and data analysis
- Instrument and data analysis intercomparisons
- Progress and actions resulting from such intercomparisons
- Absolute or relative calibration techniques
- Use of standard spectroscopic data and agreed methods for the evaluation of uncertainties
Documentation
Once an instrument is accepted, the Instrument Principal Investigator (PI) should provide and continue to update the following documentation held in the NDACC archive along with the instrument data:
- Instrument description
- Operational procedures
- Data analysis algorithm description
- Validation procedures and history
- Calibration procedures and history
Evaluation of Continuing Instruments and Instrument Teams
The Validation Appendices also provide criteria for the continuing evaluation of NDACC Instruments. The criteria address:
- Investigator responsibilities for continuing instrument verification and data analysis
- Participation in instrument and algorithm intercomparison campaigns
- Intercomparisons with measurements obtained with NDACC traveling instruments, other NDACC site instruments, instruments in other networks, and/or satellite instruments
- Absolute or relative calibration techniques and use of laboratory standards
- Use of standard spectroscopic data and agreed methods for uncertainty evaluation
- Data analysis method updates to incorporate new scientific standards
As stated in the Measurements Protocol, the experimental PI is primarily responsible for the everyday quality assurance of his/her own data, and for keeping the archives updated; however, IWGs should conduct validation exercises to the extent that resources and logistics permit. The results of continuing evaluation procedures are to be recorded in the NDACC archive and reported to the Steering Committee (SC). Measurement and analysis deficiencies found during such evaluations should be corrected, if possible, and any affected data in the archive flagged appropriately. If the PI informs the IWG Representatives that the correction of deficiencies is not presently possible, the instrument may be designated in the Measurements and Analysis Directory as currently inactive until corrective measures are possible. The PI will be notified of such a designation in writing following a decision by the NDACC SC.
Evaluation of a New Generic Measurement Method for NDACC
The evaluation procedure for the acceptance of a new generic measurement method for use within the NDACC is also addressed in the NDACC Measurements Protocol. In such cases, the Steering Committee must be provided with convincing documentation of the technique's validity, including end-to-end simulations of the measurement and data analysis methods, along with correlative experimental evidence from the actual instruments.