Knorr
IMET Data Quality Control Report
Jennifer Lovell and Shawn R. Smith
Research Vessel Surface Meteorological Data Center
Center for Ocean Atmospheric Prediction Studies
Florida State University
March 27, 2001
Report MET 01-04
Version 1.0
INTRODUCTION:
This report summarizes the quality of surface meteorological data collected by the research vessel Knorr (identifier: KCEJ) during ten cruises completed in 1993 and 1994. The data were provided to the Florida State University Data Assembly Center (DAC) in electronic format by B. Walden (WHOI) and were converted to standard DAC netCDF format. The data were then processed using an automated screening program, which added quality control flags to the data, highlighting potential problems. Finally, the Data Quality Evaluator (DQE) reviewed the data and current flags, whereby flags were added, removed, or modified according to the judgment of the DQE and other DAC personnel. Details of the quality control procedures can be found in Smith et al. (1994). The data quality control report summarizes the flags for the Knorr meteorological data, including those added by both the preprocessor and the DQE.
DATA VARIABLES:
The Knorr data are expected to include observations averaged once every minute on these cruises. Values for the following variables were collected:
|
Time Latitude Longitude Platform Course Platform Speed Over Ground Atmospheric Pressure Air Temperature Humidity Temperature Sea Temperature Relative Humidity Precipitation Atmospheric Radiation |
(TIME) (LAT) (LON) (PL_CRS) (PL_SPD) (P) (T) (T2) (TS) (RH) (PRECIP) (RAD) |
1993 FLAG SUMMARY
Details of each 1993 cruise
are listed in Table 1 and include the cruise dates, number of records, number of values,
number of flags, and total percentage of data flagged. A total of 1,924,332 values were evaluated
with 147,356 flags added by both the preprocessor and the DQE resulting in a
total of 7.66% of the values being flagged.
Table 1: Statistical Cruise Information
|
Cruise Identifier |
Cruise Dates |
Number of Records |
Number of Values |
Number of Flags |
Percent Flagged |
|
93-A 93-B 93-C 93-D 93-E 93-F 93-G |
01/28/93 – 02/01/93 02/03/93 – 02/18/93 03/18/93 – 03/28/93 07/25/93 – 08/18/93 09/27/93 – 10/21/93 11/01/93 – 11/19/93 12/08/93 – 12/21/93 |
6,788 21,697 15,138 35,290 35,525 27,097 18,826 |
81,456 260,364 181,656 423,480 426,300 325,164 225,912 |
1,890 15,719 5,017 23,292 57,084 18,240 26,114 |
2.32 6.04 2.76 5.50 13.39 5.61 11.56 |
Summary:
The 1993 IMET data from the Knorr proves to be of fair quality with 7.66% of the reported values flagged for potential problems. The distribution of flags for the remaining variables are detailed in Table 2.
Table 2: Number of Flags and Percentage Flagged for Each Variable
|
Variable |
B |
C |
F |
G |
J |
K |
S |
Total Number of Flags |
Percentage of Variable Flagged |
|
TIME LAT LON PL_CRS PL_SPD P T T2 TS RH PRECIP RAD |
3 47 1 1 1 59 54,267 |
1 |
160 159 |
76 35 1 1 |
3,101 3,623 5,624 5,982 8,457 25,097 6,338 3,997 3,746 8,885 6,223 |
1,040 1,103 519 6,742 545 69 |
71 73 154 253 93 111 21 446 21 203 7 |
1 3,332 3,855 5,781 6,282 8,551 26,325 7,498 4,963 10,569 9,633 60,566 |
0.00* 2.08 2.40 3.6 3.92 5.33 16.42 4.68 3.09 6.59 6.01 37.77 |
|
Total Number Of Flags |
54,379 |
1 |
319 |
113 |
81,073 |
10,018 |
1,453 |
147,356 |
|
|
Percent Of All Values Flagged |
2.83 |
0.00* |
1.66 |
5.87 |
4.21 |
5.21 |
7.55 |
7.66 |
|
*Percentages < 0.01
C-Flags:
During the 93-E cruise, one data value was assessed a C-flag by the preprocessor on 1993 October 2. C-flags are given to data times that are non sequential. All flags which were assessed on 1993 Jan 1 2:16, include "CFFZZBBBGGZZ". All data in the C-flagged record should not be used. The DQE felt these data values should remain in the Statistical Information for 1993 Knorr data, as the rest of the day's data are useful.
B-Flags:
Three unrealistic negative data values, –0.01, were recorded on platform course (PL_CRS).
Platform speed over ground (PL_SPD) was given 47 bounds flags to negative data values during the 93-A, 93-B, and 93-C cruises. Negative ship speed is possible when the vessel is moving at slow speeds. Currents, waves, and wind can cause realistic negative values. Negative values will also occur when the ship is in reverse.
Radiation (RAD) received 54,267 B-flags during the 1993 cruises. The values were between zero and negative one Wm-2. These physically unrealistic negative radiation values are likely the result of the instrument not tuned to low radiation values.
F-flags:
Latitude (LAT) and
longitude (LON) were assessed a total of 319 F-flags by the preprocessor during
all of the 1993 Knorr cruises.
These F-flags show that the platform speed computed by the preprocessor
exceeds the realistic speed (15 ms-1). This may have been caused by uncertainties or truncation error in
the navigation data.
G-flags:
Note: During the 93-A cruise, the ship traversed
south of 40 degrees South Latitude. In
this region of the globe, little information is known about the climatology, as
the data is sparse. Consequently, the
G-flagged data values may be realistic, though extreme observations.
Temperature (T) was assessed seven G-flags by the preprocessor during the 93-A cruise. The DQE felt these flagged values were realistic, as they were approximately four to five degrees Celsius lower than the climatological data value and were left in place to highlight extreme temperatures. During the 93-C cruise, temperature had 69 G-flags that were approximately five to six degrees Celsius higher than the climatological value; therefore, the DQE felt these were realistic, though extreme temperatures.
Humidity temperature (i.e., temperature in the relative humidity sensor) (T2) received 8 G-flags during the 93-A cruise and 27 G-flags during the 93-C cruise. The flagged values on the 93-A cruise were approximately four to five degrees Celsius lower than the climatological value. During the 93-C cruise, the G-flagged values were approximately five to six degrees Celsius higher than the climatological data value and were left in place to highlight extreme humidity temperatures.
All G-flags were left in place to highlight values that are greater than four standard deviations from the climatological mean (da Silva et al. 1994).
J-flags:
All of the J-flags assessed to the 1993 Knorr cruises were associated with measurements holding at a constant value (often zero) for an unrealistic period of time. On numerous occasions, all variables would flat-line at the same time.
K-flags:
The K-flag represents suspect data and should be used with caution. K-flags were used to highlight many different suspect situations. For example, on the last day of the 93-A cruise, temperature (T) had a three degree Celsius increase in one minute and five minutes later returned back to its previous data trend. A signature to support such an increase was not found in humidity temperature (T2) or relative humidity (RH). Consequently, the increase was flagged as suspect and should be used with caution.
Temperature (T), humidity temperature (T2), and relative humidity (RH) received K-flags on the 93-E, F, and G cruises. These K-flags emphasize periods when the three variables did not adhere to expected meteorological relationships (e.g., RH generally increases when T decreases). The exact cause of the variables' discontinuities is unknown as the DQE did not have enough meteorological (e.g., winds) or ship (e.g., heading) data to thoroughly assess the situations.
Sea temperature (TS) and precipitation (PRECIP) were given K-flags to draw attention to some type of interference that would occur on both variables at the same time. (See Fig. 1).
 |
| Fig. 1 Illustration of interference on the sea temperature and precipitation data from the 93-G Knorr cruise. |
Precipitation received K-flags for values recorded above 50 mm and for data which increased rapidly and then dropped back down to the previous data trend. (See Fig. 2).
 |
| Fig. 2 Shows precipitation data that increases rapidly and then drops back to the previous data trend. |
Radiation (RAD) received 69 B-flags on highly unrealistic data values of –600 to –400 Wm-2. These erroneous data values were changed to K-flags by the DQE.
Details of each 1994
cruise are listed in Table 3 and include cruise dates, number of records,
number of values, number of flags, and total percentage of data flagged. A total of 541,220 values were evaluated
with 24,341 flags added by the preprocessor and the DQE resulting in 4.50% of
the values being flagged.
Table 3: Statistical Cruise Information
|
Cruise Identifier |
Cruise Dates |
Number of Records |
Number of Values |
Number of Flags |
Percent Flagged |
|
94-A 94-B 94-C |
02/19/94 – 02/28/94 03/01/94 – 03/19/94 03/20/94 – 03/28/94 |
14,221 27,057 12,844 |
142,210 270,570 128,440 |
10,784 10,037 3,520 |
7.58 3.71 2.74 |
Summary:
The 1994 IMET data from the Knorr proves to be of good quality with 4.50% of the reported values flagged for potential problems. The distribution of flags for the remaining variables are detailed in Table 4.
Table 4: Number of Flags and Percentage Flagged for Each Variable
|
Variable |
B |
F |
J |
S |
Total Number of Flags |
Percentage of Variable Flagged |
|
TIME LAT LON PL_CRS PL_SPD T T2 RH PRECIP |
469 17,152 |
44 44 |
3,097 3,096 160 163 |
7 7 40 53 2 7 |
0 3,148 3,147 200 216 2 0 469 7 17,152 |
0.00 5.82 5.81 0.37 0.40 0.00* 0.00 0.87 0.01 31.69 |
|
Total Number Of Flags |
17,621 |
88 |
6,516 |
116 |
24,341 |
|
|
Percent Of All Values Flagged |
1.81 |
0.01 |
1.20 |
0.01 |
4.50 |
|
*Percentages < 0.01
Radiation (RAD) received a total 17,152 B-flags on the 1994 cruises. These values were between zero and negative one Wm-2. These physically unrealistic negative radiation values are likely the result of the instrument not tuned to low radiation values.
F-flags:
Over the course of the 1994 cruises, latitude (LAT) and longitude (LON) received 88 F-flags. These flags reveal that the platform speed computed by the preprocessor exceeds the realistic speed (15 ms-1). This may have been caused by uncertainties or truncation error in the navigation data.
J-flags:
J-flags, erroneous flags, were assigned to data for various reasons. First, J-flags were given to data that were associated with measurements holding at a constant value (often zero) for an unrealistic period of time. For example, the platform course (PL_CRS) and platform speed over ground (PL_SPD) fell out of character with the surrounding data and were consequently J-flagged.
Second, J-flags were used to highlight stair-steps in the data that were erroneous. For example, LAT, LON, PL_CRS, and PL_SPD were J-flagged when a large stair-step in the data was recorded. (See Fig. 3).
 |
| Fig. 3 Erroneous latitude, longitude, platform course, and platform speed data for the 94-C Knorr cruise. |
Removed Data:
Pressure (P) and Sea Temperature (TS) were not included in the public release of the 1994 Knorr data, because the data values were flat-lined on zero for each cruise.
Final Discussions:
Precipitation (PRECIP) data were flat-lined on zero for almost all of the 1994 Knorr cruises, except for a few spikes periodically. On the last two days of the 94-C cruise, PRECIP recorded useful data. The flat-lined data was not removed from the data set, as it was unknown to the DQE if it truly did not rain or if a possible instrument malfunction occurred and was then fixed before the end of the last cruise.
References:
Smith, S.R., C. Harvey, and D.M. Legler, 1994: Handbook of Quality Control
Procedures and Methods for Surface Meteorology Data. Report No. 141/96,
Report MET 96-1, Center for Ocean-Atmospheric Prediction Studies Florida
State University, Tallahassee FL 32306-2840
da Silva, A.M., C.C. Young and S. Levitus, 1994: Atlas of Surface Marine Data 1994,
Volume 1: Algorithms and Procedures. NOAA Atlas Series.