The NewAtmosphere plugin and atm_builder tool were created to provide a new capability supporting for physics-based atmospheric models in DIRSIG5.

Summary

The NewAtmosphere and atm_builder toolchain attempts to address the limitations of the DIRSIG4-era make_adb tool and the "Classic" atmosphere model (handled by the BasicAtmosphere plugin in DIRSIG5) combination in a variety of ways.

Improvements

  • Work with the DIRSIG5 scene HDF file directly (no need to carry around the entire scene directory).

  • Move to a binary database format that is more compact to store and faster to load.

  • Simplify the ability to integrate multiple backend physics-based atmospheric models.

  • Run multiple instances of the backend physics-based atmospheric models to build databases faster.

  • Add support for multiple times in a single database.

  • Add support for multiple altitudes in a single database.

Configuration

A detailed description of the atm_builder program is outlined in its respective manual. This section outlines the input configuration for the NewAtmosphere plugin, which is also passed to the atm_builder program to construct the database used by the plugin.

JSIM Setup Options

The inputs to the NewAtmosphere plugin and atm_builder tool are defined in a JSON document that can be either directly supplied in the JSIM file where the plugin is requested (via the inputs section) or in a separate JSON file (via the input_filename variable).

flow
Figure 1. Symbiotic Relationship of NewAtmosphere plugin and atm_builder program.

The JSON description consists of an info section that describes the atmosphere setup, a backend section that describes which backend is to be used (via the name variable) and any backed specific input variables. The role of the "backends" is to provide a unified interface for a variety of different radiative transfer (RT) models that can solve atmospheric modeling requests. For example, "get the spectral irradiance from the sun reaching a given latitude, longitude and altitude at a specific date and time". These backend adaptor modules take these generic requests and translate them into specfic inputs for the respective external atmospheric RT model (e.g. MODTRAN). This common interface means that a new atmospheric RT model can be supported once the request translation logic for that model has been written.

Note These atmospheric radiative transfer model backends are also utilized by the fourcurve_builder tool.

Using an Internal Description

Below is an example where the plugin description is fully contained in the JSIM file:

[{
  "scene_list" : [
    { "inputs" : "./demo.scene" }
  ],
  "plugin_list" : [
    {
      "name" : "NewAtmosphere",
      "inputs" : {
        "info" : {
            "name" : "A great atmosphere",
            "author" : "Joe User",
            "organization" : "Awesome, Corp",
            "description" : "This is a great atmosphere setup."
        },
        "backend" : {
            "name" : "<backend_name>",
            ...
        }
        "max_processes" : 8,
        "time_delta" : 900,
        "altitude_delta" : 500,
        "hdf_filename" : "my_sim.adb.hdf"
      }
    },
    {
      "name" : "ExampleSensorPlugin",
      "inputs" : {
        ...
      }
    }
  ]
}]

Using an External Description

Alternatively, the JSON can be stored in an external JSON file and the name of that file specified in the JSIM file. This approach allows a single copy of a configuration to be stored and shared among multiple simulations. The example below contains the plugin configuration using the same JSON structure but in an external file (named mls_rural_23km_dis4.jatm in this example):

{
    "info" : {
        "name" : "A great atmosphere",
        "author" : "Joe User",
        "organization" : "Awesome, Corp",
        "description" : "This is a great atmosphere setup."
    },
    "backend" : {
        "name" : "<backend_name>",
        ...
    }
    "max_processes" : 8,
    "time_delta" : 900,
    "altitude_delta" : 500,
    "hdf_filename" : "my_sim.adb.hdf"
}

and then this external file is referenced in the JSIM file:

[{
  "scene_list" : [
    { "inputs" : "./demo.scene" }
  ],
  "plugin_list" : [
    {
      "name" : "NewAtmosphere",
      "inputs" : {
        "input_filename" : "my_awesome_atm.jatm"
      }
    },
    {
      "name" : "ExampleSensorPlugin",
      "inputs" : {
        ...
      }
    }
  ]
}]
Note At this time, we encourage the use of the .jatm file extension for an external atmosphere description file to denote a JSON atmosphere configuration.

Common Backend Variables

There are a set of input variables that are common to all the backends available.

The maximum number of processes

The max_processes variable defines the maximum number of processes that the backend model can use. For example, multiple simultaneous MODTRAN processes can be used to generate the paths. On a multi-core computer, you can generally run one process per physical core. On a multi-core computer that supports hyper-threading, you can usually run 2 processes per physical core. The default value is set based on the number of virtual cores available on the host computer.

Note This variable can be overridden with the --max_processes command-line argument. See below for details.

The time sampling

The time_delta variable defines the time resolution of the database. Since the atmosphere is not expected to vary rapidly, it does not make sense to compute new atmospheric values at 10s, 100s or 1000s of Hz if that is what the readout rate is of the sensor. Instead, the toolchain makes the assumption that the atmosphere is optically and geometrically (sun and moon scattering) "frozen" for some period of time. The default period or time resolution is 900 seconds (15 minutes).

Note This variable can be overridden with the --time_delta command-line argument. See below for details.

The altitude sampling

The altitude_delta variable defines the altitude resolution of the database. Since the atmosphere is not assumed to be uniform in the vertical dimension, it needs to be sampled as a function of altitude if the scene(s) span significant altitude range. When the builder loads the scene(s), it looks at the altitudes spanned by the scene bounding box(es). Using the altitude sampling delta, it will construct a list of altitudes within each scene and then merge those into a combined list of altitudes for the database.

The default value for the altitude sampling delta is 500m. In most cases, the number of altitudes used in the database is a scalar multiplier for the number of backend model runs that need to be performed.

  • Decreasing the sampling delta increases the number of altitudes used in the database.

  • Increasing the sampling delta decreases the number of altitudes used in the database.

Note This variable can be overridden with the --altitude_delta command-line argument. See below for details.

The HDF database filename

The hdf_filename variable defines the name of the HDF file to hold the resulting atmospheric database. The convention so far has been to use the compound extension ".adb.hdf".

Using the Dummy Backend

The "Dummy" backend is built-in (it does not call an external program) and it implements a set simplified calculations to approximate nominal illumination conditions:

  • The sun is modeled with a 5800K blackbody spectral distribution that is magnitude normalized to approximate the Earth reaching irradiance of the sun.

  • The moon is modeled using the same solar spectral distribution, but is magnitude normalized to approximate the Earth reaching irradiance of the moon.

  • The sky is modeled with an exponentially decaying spectral distribution that approximates clear sky scattering conditions in the visible.

  • There is no path radiance or transmission along other paths.

This backed is configured without any additional options:

        "backend" : {
          "name" : "dummy"
        }
Important The "dummy" backend is primarily for testing purposes and rapid use during simulation setup and debugging. It should not be used for data production simulations.

Using the MODTRAN Tape Backend

The "MODTRAN Tape" backend is configured with a variety of options that provide details about the external MODTRAN installation and user input files. For the examples included here, the specific paths to the MODTRAN executable, data folder, etc. may differ from your installation.

Important Although the examples below show various configurations for older versions of MODTRAN, the user is strongly encouraged to use the latest version of MODTRAN6. Visit the MODTRAN page at the Spectral Sciences, Inc. website for more information.

The option to extract atmospheric profile data

The MODTRAN tape backend can optionally extract the atmospheric profile from the MODTRAN output "tape6" file. When this profile data is extracted, it is supplied to the radiometry core to drive refraction through the atmosphere. This option is controlled via the extract_profile variable, which is set to either true or false (note, boolean values in JSON are not strings, so do not assign the variable "true").

Note This option is disabled by default.

Using a MODTRAN configuration profile from your settings

The first option for specifying the MODTRAN setup is using the MODTRAN "profiles" in the user’s DIRSIG settings (or preferences). In that case the setup includes the modtran_profile variable:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_profile" : "MODTRAN5_2_4",
          "modroot_filename" : "mod5root.in",
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the profile named "MODTRAN5_2_4" might be mapped to a MODTRAN5 2.4 installation.

Note This backend uses the "modroot" approach to naming the input and output files associated with the MODTRAN run (see the MODTRAN documentation for more information about this mechanism). The modroot_filename variable indicates the name of the file containing the base name of the MODTRAN input and output files. Since the name of this file has changed depending on the version of MODTRAN, it is provided as a user configuration variable.

Supplying the installation paths: MODTRAN4

Another option is to specify the path to the executable and data directory directly in the backend description:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/Mod4v3r1/Mod4v3r1.exe",
          "modtran_data" : "/Users/dirsig/Mod4v3r1/Mod4v3r1/DATA",
          "modroot_filename" : "modroot.in",
          "modtran_min_delta" : 1.0,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the user is pointing at a MODTRAN4 Version 3, Release 1 installation in their own directory. The modtran_exe variable is assigned the full path to the MODTRAN executable, the modtran_data is assigned the full path to the MODTRAN DATA directory and the modtran_min_delta variable is assigned the limiting frequency resolution for the band model database available with that version of MODTRAN (typically 1.0 wavenumbers for MODTRAN4). The name of the "modroot" file for MODTRAN4 was modroot.in.

Supplying the installation paths: MODTRAN5

Another option is to specify the path to the executable and data directory directly in the backend description:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/Modtran5.2.4/mod5_cons.exe",
          "modtran_data" : "/Users/dirsig/Modtran5.2.4/DATA",
          "modroot_filename" : "mod5root.in",
          "modtran_min_delta" : 0.1,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }

In this example, the user is pointing at a MODTRAN5 2.4 installation in their own directory. The modtran_exe variable is assigned the full path to the MODTRAN executable, the modtran_data is assigned the full path to the MODTRAN DATA directory and the modtran_min_delta variable is assigned the limiting frequency resolution for the band model database available with that version of MODTRAN (0.1 wavenumbers for MODTRAN5 and later). The name of the "modroot" file for MODTRAN5 was mod5root.in.

Note The specific paths to the MODTRAN executable will vary based on your installation.

Supplying the installation paths: MODTRAN6

Although MODTRAN6 introduced a new JSON input/output mechanism, the program is backward compatible with the traditional "tape" input/output mechanism utilized by this backend. This input/output option is accessed via the -modroot command line option, which is described in the manual and shown in the usage message of the console executable:

$ ./mod6c_cons -h
Usage: ./mod6c_cons input_file [data_path]
JSON and TAPE5 input formats are supported.
[data_path] specifies an optional path to the MODTRAN data directory.

optional flags
  -modroot
   Process legacy input formats using mod5root.in or tape5 files.

...

This backend employs the "modroot" approach for all versions of MODTRAN, where the basename of the input/output files for the MODTRAN run is stored in a separate file. MODTRAN5 and MODTRAN4 automatically looked for the mod5root.in and modroot.in files, respectively. MODTRAN6, however, needs the -modroot option to be specified as the 1st command-line argument for it to look in the mod5root.in file. Additionally, the MODTRAN6 executable needs to be supplied the full path to the MODTRAN6 DATA folder as the 2nd command-line argument. Hence, the complete command-line to launch MODTRAN6 needs to look like:

$ /some/path/MODTRAN6.0.0/bin/mod6c_cons.exe -modroot /some/path/MODTRAN6.0.0/DATA

This backend has an optional configuration variable that allows the user to customize the command-line arguments that are passed to the MODTRAN executable. These arguments are defined in the JSON using the modtran_options variable, which is a JSON array of strings. The example configuration below shows how these two additional arguments are provided to the MODTRAN executable using the options array variable:

        "backend" : {
          "name" : "modtran_tape",
          "modtran_exe" : "/Users/dirsig/MODTRAN6.0.0/mod6c_cons.exe",
          "modtran_data" : "/Users/dirsig/MODTRAN6.0.0/DATA",
          "modroot_filename" : "mod5root.in",
          "modtran_options" : ["-modroot","/Users/dirsig/MODTRAN6.0.0/DATA"],
          "modtran_min_delta" : 0.1,
          "tape5_filename" : "./mls.tp5",
          "extract_profile" : false
        }
Note The specific paths to the MODTRAN executable will vary based on your installation.

The input "tape5" filename

The tape5_filename variable defines the name of the input "tape5" file that is used as a template for the MODTRAN runs. In general, the backend is using the description of the atmosphere in this file (which model, aerosols, visibility, etc.) and is only updating the key parameters that are relevant to the specific DIRSIG simulation (the wavelengths, the source position, the path geometry, etc.).

The user also has the option to store the MODTRAN "tape5" file within the input JSON description via the tape5_contents variable. However, because the "tape5" file is multiple lines and you cannot embed newlines inside a JSON variable, the user will need to make the file a single string by replacing the newlines with the \n character:

    "name" : "NewAtmosphere",
    "inputs" : {
      "info" : {
        [deleted for documentation purposed]
      },
      "backend" : {
        "name" : "modtran_tape",
        "modtran_profile" : "MODTRAN4",
        "modtran_exe" : "/Users/dirsig/pkg/Mod4v3r1/Mod4v3r1.exe",
        "modtran_options" : [""],
        "modtran_data" : "/Users/dirsig/pkg/Mod4v3r1/DATA",
        "extract_profile" : false,
        "tape5_contents" : "M   2    2    2    0    0    0    0    0    0    0    1    1    1   0.000   0.00\n    2    0    0    1    0    0     0.000     0.000     0.000     0.000     0.218\n    20.218     0.218   180.000     0.000     0.000     0.000    0\n    1    2  236    0\n    43.000    77.000     0.000     0.000    12.000     0.000     0.000     0.000\n       450     35050       100     2\n0"
      },
      "hdf_filename" : "mls_rural_23km_dis4_scaled.adb.hdf"
    }
Important The user needs to be very careful when performing this newline replacement operation to otherwise not disrupt the precise format of the MODTRAN "tape5" file.

Errors and Debugging

In most situations, the source of errors will be related to the external model executed by the backend. When an error occurs with either the running of the backend model process or in reading the output of the backend model, an error will be generated and the related files will be preserved in the reported path for debugging.

Under the hood of the MODTRAN Tape backend

The "MODTRAN tape" backend creates a folder for all the MODTRAN executions (named modtran_runs) within which each model execution is stored in a sub-folder using a simple numerical numbering scheme from 0 to N. The backend will purge the contents of the model request folders after the model has been successfully run and the output ingested. However, these folders can be preserved by using the --save_files command-line option to the builder.

The file listing below shows the contents of a preserved model request folder after execution:

$ ls modtran_runs/0
README.txt              modtran_stderr.txt      tmp.7sc
mod5root.in             modtran_stdout.txt      tmp.tp5

The README.txt file contain a "human readable" description of the inputs to MODTRAN. The mod5root.in contains the basename of the input/output files (in the case of this backend, it is always tmp), the modtran_stderr.txt and modtran_stdout.txt files contain the standard error and standard output (console) of the MODTRAN program, respectively. The tmp.tp5 and tmp.7sc files are the respective input and output files of the model execution.