Introduction
Intercomparisons are essential elements in NDACC instrument certifications. This document describes the method recommended by the NDACC Steering Committee for formal campaign-style instrument intercomparisons leading to their validation and certification for use at NDACC stations. It describes the roles of the various parties participating in the intercomparison and the rules that should be followed for a successful exercise (i.e., one that is accepted by the scientific community as being objective and thorough).
The formal instrument intercomparison could be preceded by, accompanied by, or followed by formal or informal data analysis intercomparisons, since full certification involves the intercomparison of the instruments and their associated analysis procedures. In addition, an informal instrument intercomparison that precedes a formal one could avoid the situation of formal (first-time) intercomparisons resulting in little communication among investigators and an insufficient learning experience. Thus, a sequence of informal campaigns followed by a formal campaign could be considered the natural evolution in the development of measurement systems.
Certainly, more frequent informal intercomparisons and collaborations are encouraged at any time.
Goal
Decisions on instrument certification require that instrument intercomparisons be aligned to established measurement (retrieval) goals. For most exercises, this goal can be stated as: to intercompare the named atmospheric species or parameters measured simultaneously by participants who do not see each other's results during the campaign, and whose data are submitted on a prescribed timetable to a referee during the course of the campaign.
Participation
Two levels of participation in a formal campaign are possible:
Formal – The participating individual or group agrees to have its results published as submitted by the final submission date. This is the recommended mode for NDACC certification.
Informal / Learning Experience – No results from the participating individual or group are published, and discussions with groups at the site are permitted. Such participation enables a new group to learn and to evolve within the measurement community prior to its participation in an exercise leading to NDACC certification.
Results
The results are presented to the scientific community by an impartial referee who formalizes and monitors the campaign. The referee is an on-site participant in the campaign, and has authority over its activities. The referee collects the data at preset times for comparison and analysis, and prepares the data for presentation at a participant workshop and to the NDACC Steering Committee as well as for possible publication in a refereed journal or report.
Intercomparison Definition and Structure
Definitions
- Organizers: They are typically members of the NDACC Instrument Working Groups associated with the instrument types undergoing formal intercomparison. They bring together a number of measurement groups for the intercomparison on behalf of the Steering Committee.
- Referee: This individual has the responsibility for handling all of the data during the intercomparison campaign, and is the decision-maker in charge of its conduct. This person is responsible for insuring that the intercomparison is “blind”.
Structure
- The organizers, in cooperation with the referee, must specify in detail prior to the beginning of the campaign the species / parameters to be measured, the method(s) to be used, the times of operation, the data formats, and the schedule for data submission both during and after the campaign.
- The organizers, in cooperation with the referee, must specify in detail the calibration techniques to be followed during the intercomparison, and the frequency and method of their use. This responsibility includes the collection of instrument calibration requirements prior to the campaign so that the calibration method can be used effectively during the campaign and an effective plan for reaction to the calibration results can be implemented. This will enable a true on-site comparison of results by the referee.
- Prior to the campaign, each participant must submit to the referee a detailed description of the instrument and the analysis technique. The instrument should not be changed during the campaign.
- Participants must also submit a detailed description of instrumental parameters such as spectral ranges, spectral lines, etc. to be used to insure measurement comparability by every group.
- The impartiality of any (formal) intercomparison must not be compromised. Thus, an investigator should only be able to see other data sets prior to the submission of the final data if they are made available in an anonymous manner (i.e./, without attribution to the measurement group). The benefits of investigator interaction at an observing site before and during a campaign may be acquired via informal exercises, as discussed earlier.
Referee's Role
- To insure as far as possible that an impartial and blind intercomparison is achieved.
- To be unbiased and tactful, but tough and decisive, when necessary.
- To coordinate the observations to insure simultaneity, and to maximize equality among the participating instruments.
- To recognize observing or data analysis practices that could introduce differences in the results that are not primarily due to instrumental differences, and to advise the affected group(s) accordingly.
- To mediate and, as far as possible, resolve problems, and to hold discussions with all participants as the needs arise.
- To mediate and resolve problems that arise following the final data submission date (i.e., with investigators who may want to change data or otherwise modify their results).
- To record all instructions to participants, and keep a good log of observations.
- To examine the primary data sets quickly, and to advise only those participants whose results are markedly different from the overall results. This avoids trivial errors spoiling a group's contribution for more than a day or two; however, this option must be exercised with caution.
- To be on-site during the campaign. This encourages impartiality and helps insure to the community that the campaign was conducted properly.
- To coordinate participation in a post-campaign workshop that is open to the community.
- To coordinate the publication of the results in a refereed journal or report. All participants are to review the report prior to publication, and to be included as authors of this publication.
- To coordinate the campaign with the organizers at the chosen site.
- To organize meetings as required during the campaign.
- To maintain the master clock for synchronization of the data.
Data Submission
- Data should be turned in to the referee on a regular, predefined basis (typically 24 hours after being taken) so the referee can begin to compile and review the results. However, participants could be allowed (at the discretion of the referee) to change their data during the campaign in response to instrumental findings or measurement. This possibility recognizes that field campaigns are stressful, that equipment may arrive damaged, and that mistakes may be made while moving into an operational mode in the field.
- A final data submission date should be set prior to the start of the campaign. Participants can change their data prior to this final date and after the closing of the campaign, but they must submit an acceptable explanation to the referee. This explanation might be published at the discretion of the referee as part of the campaign paper. This allows for the correction of obvious mistakes and for a final tuning of the results. It is recommended that the final submission date be no more than six weeks after the closing of the campaign.
- As mentioned above, the results from any other group may only be seen by any participant prior to the release of the data at the workshop if the results are available without attribution to a specific measurement group.
Exceptions
For some intercomparison campaigns, ancillary data taken at the campaign site (or elsewhere) may be required for proper data analysis. In such cases, the organizers/referee will determine the optimal schedule for data submission.
The length of some campaigns may make it impossible for the referee to be onsite for the duration. Should this be the case, a plan should be in place to prevent compromising the blindness of the campaign.
For some campaigns, it may be desirable that final data be submitted at a workshop held at the end of the campaign. In such a case, no subsequent changes in data will be allowed; otherwise the impartiality of the intercomparison would be compromised.
Auxiliary Data
Prior to the campaign, the organizers/referee should determine any auxiliary data that are required and should invite the appropriate persons to provide these data.
Post-Campaign Workshop
The referee should organize a workshop that is open to the community several months after the campaign. Ample opportunity should be provided for the participants to present and discuss the campaign results.
Future Instrument Validation
NDACC-approved instruments may be used to certify new instruments that become available at the same site at which the certified instrument is operating. This recognizes that major intercomparison campaigns are expensive, time-consuming, and require participation by many members of the research community. Thus, it will be difficult to conduct them on a frequent basis. An NDACC Instrument Working Group can also make use of a "certified" traveling instrument for validation. In such cases, the Working Group should specify the testing approach that leads to the certification of the traveling instrument.
Instrument Specific Requirements
Instrument specific requirements associated with the validation and intercomparison of instruments within NDACC are provided in the appendices to the NDACC Validation Protocol.