How DART stores observations: observation sequence (obs_seq) files
Since DART is designed to assimilate observations from any data source, it
includes a set of programs to convert observations from their original
format to DART’s own observation sequence, or obs_seq, format. The
obs_seq format is designed to allow DART to accomodate a myriad of source
observation file formats, structure and metadata. Many original source
observation files don’t contain the necessary information about the error
characteristics and spatial structure of the data needed to perform an
assimilation.
There are three types of obs_seq files.
obs_seq.in
An obs_seq.in file actually contains no observation quantities. It may be
best thought of as a perfectly laid-out notebook waiting for an observer
to fill in the actual observation quantities.
All the rows and columns are ready, labelled, and repeated for every
observation time and platform. The obs_seq.in file is generally the start
of a “perfect model” experiment.
In a perfect model experiment, one instance of the model is run through the
DART program perfect_model_obs - which applies the appropriate forward
operators to the model state and writes down the observations generated by the
model in the writes them down in the perfectly laid-out notebook.
The completed notebook is then renamed obs_seq.out.
obs_seq.out
An obs_seq.out file contains a linked list of observations. The
observations can potentially be (and usually are) from different platforms and
of different quantities, each with their own error characteristics and
metadata.
An obs_seq.out file containing real data can be generated by using one of
DART’s many observation converter programs. Additionally, an obs_seq.out
file containing synthetic data can be created by running DART’s
perfect_model_obs program.
The observations in the obs_seq.out files are assimilated into the model
ensemble by DART’s filter program.
To learn more about the structure of the obs_seq.out file, see
Detailed structure of an obs_seq file.
If you want to create an observation sequence file from real observations, you should contact DAReS staff by emailing dart@ucar.edu for advice regarding your specific types of observations.
obs_seq.final
When running an assimilation, DART’s filter program assimilates the
observations contained in the obs_seq.out file and generates an
obs_seq.final file.
The obs_seq.final file contains everything in the obs_seq.out file and
also contains a few additional ‘copies’ of the observation.
Since DART is an ensemble algorithm, each ensemble member must compute its own
estimate of the observation for the algorithm. You can save the ensemble
members’ estimates of the observation in the obs_seq.final file by setting
the num_output_obs_members entry in the filter_nml namelist of
input.nml to a value greater than zero.
Minimally, filter will record the mean and spread of the ensemble estimates
in the obs_seq.final file.
To learn more about the structure of the obs_seq.final file, see
Detailed structure of an obs_seq file.
Using obs_seq.final for observation-space diagnostics
The best method to determine the performance of an experiment in which you assimilate data from real-world sources is to compare the ensemble estimates of the observation to your real-world data. You can estimate the bias and error of the ensemble mean or gauge how many of the real-world observations are actually being assimilated. These diagnostics are known as observation-space diagnostics.
DART provides programs obs_diag and MATLAB observation space diagnostics for you use to quickly assess the performance of your experiment.
Note
Since each ‘observation type’ may require different amounts of metadata to
be read or written, any routine to read or write an observation sequence
must be compiled with support for those particular observations. The
supported observations are listed in the obs_kind_nml namelist of
input.nml. For more information, see How DART supports different types of observations: the preprocess program.