External (precomputed) forward operators

Overview

In data assimilation the process of computing an observation’s expected value from a model state is called the forward operator (FO). Normally filter computes the expected values by calling the forward operator code, internal to DART, for each observation, which applies appropriate conversions from fundamental model state variables, followed by a spatial interpolation to the observation location. Some systems compute the expected observation values outside of DART, for example:

  • Another data assimilation system, such as NOAA’s Gridpoint Statistical Interpolation (GSI) system, has already computed expected values for radiances or other observations that require expensive forward operator calculations,

  • The model itself applies its own forward operator as it advances, such as ROMS, which produces its own ‘verification’ observations,

  • The forward operator requires model-specific fields or capabilities which are not available in the DART state vector.

In these cases, an ensemble of expected observation values can be stored in the observation sequence file along with each observation. These values are called external or precomputed forward operators. When filter is told to use them (through the &obs_kind_nml namelist, described below) it reads the ensemble of expected values from the observation sequence file and uses these external forward operator values directly instead of calling forward operator code.

Note: Internal vs External Forward Operators

External (precomputed) forward operators are values computed by a program external to DART and provided to DART through an observation sequence. Internal forward operators are calculated from the state in DART.

  • Observations generated by the program perfect_model_obs are not external forward operators.

  • Forward operators calculated using the Optional Precomputed Fields technique described in Forward Operators, where a model computes an extra diagnostic field that is added to the DART state vector and interpolated by the forward operator, code are not external forward operators.

  • Forward operators that DART calculates by calling a library, such as RTTOV, are not external forward operators.

There are three steps to using external forward operators:

  1. Create an observation sequence file containing the precomputed forward operator values.

  2. List the observation types that should use the precomputed values in the use_precomputed_FOs_these_obs_types item of the &obs_kind_nml namelist.

  3. Run filter.

Warning

External forward operators are calculated from the ensemble of model states outside the filter program, that is before inflation is applied in filter. If you are using inflation in filter, the precomputed values will not be consistent with the inflated ensemble.

Creating an obs_seq file with precomputed forward operators

There are two ways to create an observation sequence file containing precomputed forward operator values, either

  • Use an observation converter that writes the values along with the observations, or

  • Add the values to an existing observation sequence file with pyDARTdiags.

Using an observation converter

Two converters in the DART distribution write precomputed forward operator values and can be used as examples:

  • GSI2DART converts GSI diagnostic files, which contain the expected values computed by GSI for each ensemble member, into a DART observation sequence file.

  • convert_roms_obs converts ROMS verification observation files, one per ensemble member, into a DART observation sequence file.

If you are writing your own converter, attach the ensemble of expected values to each observation with the set_obs_def_external_FO routine from obs_def_mod before adding the observation to the sequence. For example, from convert_roms_obs.f90:

use obs_def_mod, only : set_obs_def_external_FO, obs_def_type

...

call get_obs_def(obs, obs_def)
call set_obs_def_external_FO(obs_def, .true., .true., key, ens_size, expected_obs)
call set_obs_def(obs, obs_def)
call add_obs_to_seq(obs_seq, obs, time_obs, prev_obs, prev_time, first_obs)

The arguments to set_obs_def_external_FO are:

subroutine set_obs_def_external_FO(obs_def, has_external_FO, write_external_FO, &
                                   external_FO_key, ens_size, external_FO_values)

type(obs_def_type), intent(inout) :: obs_def
logical,            intent(in)    :: has_external_FO   ! this obs has precomputed values
logical,            intent(in)    :: write_external_FO ! write the values to the output file
integer,            intent(in)    :: external_FO_key   ! identifier, useful for debugging
integer,            intent(in)    :: ens_size          ! number of precomputed values
real(r8),           intent(in)    :: external_FO_values(ens_size) ! one value per ensemble member

Using pyDARTdiags

If you already have an observation sequence file and an ensemble of expected observation values, you do not need to write an additional Fortran observation converter. pyDARTdiags, a Python library for working with DART observation sequence files, can read an existing obs_seq file, attach an ensemble of precomputed forward operator values to each observation, and write out a new obs_seq file ready to be used by filter.

See the pyDARTdiags example notebook Adding external forward operator values to an existing obs_seq file for an example workflow.

File format

In an ASCII observation sequence file the precomputed forward operator values are stored as an external_FO record after the observation type (and any type-specific metadata) and before the observation time. The external_FO line contains the number of values (the ensemble size) and a key, and is followed by the values themselves - one expected value per ensemble member, in ensemble member order. The key is stored but not used by filter; it is only an identifier for debugging and does not have to match the observation number. Note when using binary observation sequence files the key is discarded and replaced with a local counter.

 OBS            1
   0.135100000000000
   0.000000000000000
          -1           2          -1
obdef
loc3d
     4.712388980384690        0.6283185307179586        -500.0000000000000     3
kind
          60
external_FO       3       1
   0.131100000000000        0.135200000000000        0.129800000000000
     43200    151945
   1.000000000000000E-004

In this example the observation has 3 precomputed forward operator values, one for each of 3 ensemble members. See Detailed structure of an obs_seq file for a description of the rest of the observation sequence file format.

Running filter

To make filter use the precomputed values instead of calling the forward operator code, list the observation types in the use_precomputed_FOs_these_obs_types item of the &obs_kind_nml namelist in input.nml. Each type must also be listed in either assimilate_these_obs_types or evaluate_these_obs_types, which control whether the observations are assimilated or evaluated only. For example:

&obs_kind_nml
   assimilate_these_obs_types          = 'SATELLITE_SSH',
                                         'ARGO_TEMPERATURE',
                                         'RADIOSONDE_TEMPERATURE'
   use_precomputed_FOs_these_obs_types = 'SATELLITE_SSH',
                                         'ARGO_TEMPERATURE'
  /

With this namelist, the SATELLITE_SSH and ARGO_TEMPERATURE observations are assimilated using the precomputed forward operator values from the observation sequence file, while the RADIOSONDE_TEMPERATURE observations are assimilated by calling the forward operator code as usual.

Setting use_precomputed_FOs_these_obs_types = 'all' uses the precomputed values for every type listed in assimilate_these_obs_types and evaluate_these_obs_types.

See the obs_kind_mod namelist documentation for the full description of &obs_kind_nml.

Requirements

  • External forward operators are calculated from your forecast ensemble, so must be added to your cycling workflow.

  • Every observation of a type listed in use_precomputed_FOs_these_obs_types must contain precomputed values in the observation sequence file. It is a fatal error if filter needs a precomputed value that is not present.

  • The number of precomputed values stored with an observation must be at least as large as the ensemble size used by filter. It is a fatal error if there are fewer values than ensemble members.

  • The precomputed values are matched to the ensemble members in order: the first value is used for ensemble member 1, the second for member 2, and so on. It is up to you to ensure the order of the values in the observation sequence file is consistent with the order of the ensemble members given to filter.

  • Conversely, if an observation has precomputed values but its type is not listed in use_precomputed_FOs_these_obs_types, the values are ignored and the forward operator code is called as usual.

Posterior forward operators and DART quality control

The precomputed values are prior forward operators - they were computed from the ensemble of model states before assimilation. filter can neither compute nor ingest a posterior forward operator for these observations: the internal forward operator code is not being used, and the precomputed values in the observation sequence file are used only for the prior. There is no way to supply posterior values through the observation sequence. If posterior values are requested (compute_posterior = .true. in &filter_nml), the posterior expected values for these observations are reported as missing, and the DART quality control value is set to 2 (assimilated successfully, but the posterior forward operator failed) or 3 (evaluated only, posterior forward operator failed).

If you need posterior expected values for these observations, compute them outside DART by applying your external forward operator to the analysis ensemble written by filter.

All other DART quality control settings behave as usual; for example, an observation rejected by the outlier threshold still gets a DART QC of 7. See MODULE quality_control_mod for the complete list of DART QC values.

Note that the obs_seq.final file written by filter does not contain the external_FO metadata records. If you request that ensembles of prior expected values are output (num_output_obs_members in &filter_nml), the precomputed values appear as the prior ensemble member copies of the observations, just as they would for forward operators computed by filter.

Manipulating obs_seq files with precomputed forward operators

By default the program obs_sequence_tool passes precomputed forward operator values through to the output file just like any other observation metadata, so you can subset and combine observation sequence files as usual. If you want to remove the precomputed values (and keep the observations), use the remove_precomputed_FO_values namelist item of &obs_sequence_tool_nml:

# keep all precomputed values (the default):
   remove_precomputed_FO_values = ''

# remove all precomputed values from all observations:
   remove_precomputed_FO_values = 'ALL'

# remove the precomputed values for specific observation types only:
   remove_precomputed_FO_values = 'SATELLITE_SSH', 'ARGO_TEMPERATURE'