Introduction
Intercomparisons are essential elements in NDACC instrument and measurement certifications. This document describes the method recommended by the NDACC Steering Committee for formal campaign-style instrument intercomparisons leading to their validation and certification for use at NDACC stations. It describes the roles of the various parties participating in the intercomparison and the rules that should be followed for a successful exercise (i.e., one that is accepted by the scientific community as being objective and thorough).
The formal instrument intercomparison could be preceded by, accompanied by, or followed by formal or informal data analysis intercomparisons, since full certification involves the intercomparison of the instruments and their associated data analysis procedures. In addition, an informal instrument intercomparison that precedes a formal one could avoid the situation of formal (first-time) intercomparisons resulting in little communication among investigators and an insufficient learning experience. Thus, a sequence of informal campaigns followed by a formal campaign could be considered the natural evolution in the development of measurement systems.
Certainly, more frequent informal intercomparisons and collaborations are encouraged at any time.
Goal
Decisions on instrument certification require that instrument intercomparisons be aligned to established measurement (retrieval) goals. For most exercises, this goal can be stated as: to intercompare the named atmospheric species or parameters measured simultaneously by participants who do not see each other's results during the campaign, and whose data are submitted on a prescribed timetable to a referee during the campaign.
Participation
Two levels of participation in a formal campaign are possible:
- Formal – The participating individual or group agrees to have its results published as submitted by the final submission date. This is the recommended mode for NDACC certification.
- Informal / Learning Experience – No results from the participating individual or group are published, and discussions with groups at the site are permitted. Such participation enables a new group to learn and to evolve within the measurement community prior to its participation in an exercise leading to NDACC certification.
Results
The results are presented to the scientific community by an impartial referee who formalizes and monitors the campaign. The referee is an on-site participant in the campaign and has authority over its activities. The referee collects the data at preset times for comparison and analysis, and prepares the data for presentation at a participant’s workshop and to the NDACC Steering Committee, as well as for possible publication in a refereed journal or report.
Intercomparison Definition and Structure
Definitions
- Organizers: They are typically members of the NDACC Instrument Working Groups associated with the instrument types undergoing formal intercomparison.They bring together a number of measurement groups for the intercomparison on behalf of the Steering Committee.
- Referee: This individual has the responsibility for handling all the data during the intercomparison campaign and is the decision-maker in charge of its conduct. This person is responsible for ensuring that the intercomparison is impartial.
Structure
- The organizers, in cooperation with the referee, must specify in detail prior to the beginning of the campaign the species / parameters to be measured, the method(s) to be used, the times of operation, the data formats, and the schedule for data submission both during and after the campaign.
- The organizers, in cooperation with the referee, must specify in detail the calibration techniques to be followed during the intercomparison, and the frequency and method of their use. This responsibility includes the collection of instrument calibration requirements prior to the campaign so that the calibration method can be used effectively during the campaign and an effective plan for reaction to the calibration results can be implemented. This will enable a true on-site comparison of results by the referee.
- Prior to the campaign, each participant must submit to the referee a detailed description of the instrument and the analysis technique. The instrument should not be changed during the campaign without notification and approval of the referee and other participants.
- Participants must also submit a detailed description of instrumental parameters such as spectral ranges, spectral lines, etc. to be used to ensure measurement comparability by every group.
- The impartiality of any (formal) intercomparison must not be compromised. Thus, an investigator should only be able to see other data sets prior to the submission of the final data if they are made available in an anonymous manner (i.e., without attribution to the measurement group). The benefits of investigator interaction at an observing site before and during a campaign may be acquired via informal exercises, as discussed earlier.
Referee's Role
- To coordinate the campaign with the organizers at the chosen site.
- To ensure as far as possible that an impartial intercomparison is achieved.
- To be unbiased and tactful, but tough and decisive, when necessary.
- To coordinate the observations to ensure simultaneity, and to maximize equality among the participating instruments.
- To maintain the master clock for synchronization of the data.
- To recognize and account for observing or data analysis practices that could introduce differences in the results that are not primarily due to instrumental differences.
- To mediate and, as far as possible, resolve problems that arise during the campaign, and to hold discussions with all participants as needed.
- To mediate and resolve problems that arise following the final data submission date (i.e., with investigators who may want to change data or otherwise modify their results).
- To record all instructions to participants and keep a good log of observations.
- To examine the primary data sets quickly, and to advise only those participants whose results are markedly different from the overall results. This avoids trivial errors spoiling a group's contribution for more than a day or two; however, this option must be exercised with caution.
- To be on-site during the campaign. This encourages impartiality and helps ensure to the community that the campaign was conducted properly.
- With the organizers to organize meetings as required during the campaign.
- With the organizers to coordinate participation in a post-campaign workshop that is open to the community.
- With the organizers to coordinate the publication of the results in a refereed journal or report. All participants are to review the report prior to publication, and to be included as authors of this publication.
Data Submission
- Data should be turned in to the referee on a regular, predefined basis (typically 24 hours after being taken) so the referee can begin to compile and review the results. However, participants could be allowed (at the discretion of the referee) to change their data during the campaign upon detection of instrumental or measurement problems. This possibility recognizes that field campaigns are stressful, that equipment may arrive damaged, and that mistakes may be made while moving into an operational mode in the field.
- A final data submission date should be set prior to the start of the campaign. Participants can change their data prior to this final date and after the closing of the campaign, but they must submit an acceptable explanation to the referee. This explanation might be published at the discretion of the referee as part of the campaign paper. This allows for the correction of obvious mistakes and for a final tuning of the results. It is recommended that the final submission date be no more than six weeks after the closing of the campaign.
- As mentioned above, the results from any other group may only be seen by any participant prior to the release of the data at the workshop if the results are shown without attribution to a specific measurement group.
Exceptions
For some intercomparison campaigns, ancillary data taken at the campaign site (or elsewhere) may be required for proper data analysis. In such cases, the organizers/referee will determine the optimal schedule for data submission.
The length of some campaigns may make it impossible for the referee to be on-site for the duration. Should this be the case, a plan should be in place to prevent compromising the impartiality of the campaign.
For some campaigns, it may be desirable that final data be submitted at a workshop held at the end of the campaign. In such a case, no subsequent changes in data will be allowed; otherwise, the impartiality of the intercomparison would be compromised.
Auxiliary Data
Prior to the campaign, the organizers/referee should determine any auxiliary data that are required and should invite the appropriate persons to provide these data.
Post-Campaign Workshop
The referee should organize a workshop that is open to the community several months after the campaign. Ample opportunities should be provided for the participants to present and discuss the campaign results.
Future Instrument Validation
NDACC-approved instruments may be used to certify new instruments that become available at the same site at which the certified instrument is operating. This recognizes that major intercomparison campaigns are expensive, time-consuming, and require participation by many members of the research community. Thus, it will be difficult to conduct them on a frequent basis. An NDACC Instrument Working Group can also make use of a "certified" traveling instrument for validation. In such cases, the Working Group should specify the procedure that leads to the certification of the traveling instrument.
Instrument Specific Requirements
Instrument specific requirements associated with the validation and intercomparison of instruments within NDACC are provided in the appendices to the NDACC Validation Protocol.
Current online Version: November 14, 2025
Revision: November 14, 2025
Renamed document from ‘NDACC Protocol for Instrument Intercomparisons’ to ‘NDACC Protocol for Instrument Intercomparisons and Campaigns’. Removed remaining ‘blind’ references. Clarified some roles of organizer vs. referee.
Revision: March 9, 2017
Some wording changes. Addition of items in the Structure section on determination of calibration techniques. The intercomparison standard of ‘blind’ is now ‘impartial’.
Revision: March 24, 2009
Name of the network is updated to NDACC.
Original Version: September 9, 1996