This guide outlines the command-line interface to the DIRSIG5 model.

Scene Compilation

In order to run a DIRSIG simulation, the scene must be converted into a pre-optimized form that the DIRSIG5 model will ingest. This conversion process is referred to as "scene compilation" and was introduced to facilitate faster start-up times with DIRSIG5. With DIRSIG4, the loading of the scene geometry could be a very slow process if the scene was large and contained a lot of geometry. This DIRSIG4 compute time was spent performing sanity checks on the scene geometry, the association of geometry to materials, etc. The realization was that this work was repeated at the start of every simulation, even if the scene had not changed. With DIRSIG5, we replaced this frequent and compute intensive task with an infrequent pre-simulation task. The scene2hdf program included with DIRSIG5 will "compile" a DIRSIG4 scene into a single, binary Hierarchical Data Format (HDF) file that can be quickly loaded and utilized by the DIRSIG5 radiometry simulation. This compilation step performs all of these checks on the geometry and materials once and then allows the DIRSIG5 model to use it many times. For large, complex scenes this scene load process could take several (sometimes 10s) of minutes in DIRSIG4 whereas the DIRSIG5 scene load takes a few seconds.

Important The user does not need to compile the scene before every simulation. This process only needs to be repeated if the contents of the scene have been modified.

To run the scene2hdf scene compiler, you provide the name of the scene file to the program:

$ scene2hdf my_site.scene
Note Please consult the scene2hdf manual to discover and understand all the options to the scene compiler.

The scene compiler will then begin the process of reading and checking all the geometry. It will then produce a file with the same name as the input .scene file but with .hdf appended to it. From our example above, the name of the final scene HDF file would be my_site.scene.hdf.

Important The HDF5 library expects the filesystem to support file locking to avoid corruption introduced by simultaneous read/write commands. If your filesystem doesn’t support file locking, then the HDF5 library will issue an error. The user can disable HDF5 file locking by setting the environment variable HDF5_USE_FILE_LOCKING to FALSE.

When the tool completes, it will print a series of warnings and (potentially) errors. Most warnings are for currently unsupported features that the tool can work around. Errors arise in situations where the tool cannot find a reasonable work around. Please consult the release notes document for each beta release for discussions of currently unsupported features and previously unsupported features that are now supported.

Running a Simulation

Using a DIRSIG4 Input Description

DIRSIG5 comes with a compatibility layer that allows it to read DIRSIG4 era .sim files, which will automatically trigger the configuration of the DIRSIG4 compatibility plugins to support those older configurations. This compatibility mode is automatrically triggered by the detection of the .sim file extension on the supplied input file:

$ dirsig5 test.sim
Tip The user can emulate the DIRSIG4 "preview mode" by supplying the -preview option to DIRSIG5. However unlike DIRSIG4, which would skip radiance calculations in this mode, the DIRSIG5 simulation will produce a low-fidelity radiance solution.

DIRSIG5 also supports the DIRSIG4 "component mode" syntax that allowed the user to run a simulation without a DIRSIG4 .sim file by supplying the list of files that would otherwise be captured in the .sim file:

$ dirsig5 --scene_filename=foo.scene --atm_filename=foo.atm \
    --platform_filename=foo.platform --motion_filename=foo.ppd \
    --tasks_filename=foo.tasks

This mode in DIRSIG5 also supports the options to specify tasks as command-line options rather than via a .tasks file. See the component mode documention in the DIRSIG4 Command-Line Guide for the complete overview of options.

Using a DIRSIG5 Input Description

DIRSIG5 introduced the new JSIM input description. This input gives the user directy access to the plugins used in the model execution and the inputs to those plugins. The native DIRSIG5 execution mode is automatrically triggered by the detection of the .jsim file extension on the supplied input file:

$ dirsig5 test.jsim

Options

Controlling Fidelity

In DIRSIG5, the fidelity (quality) of the simulation is driven by how many paths are used in each pixel and the number of nodes (bounces) in those paths. The current algorithm supports an adaptive sampling scheme (similar to the optional adaptive mode employed in DIRSIG4) to vary the number of paths (samples) used in each pixel based on the complexity of the content within that pixel. A pixel with a larger dynamic range, more sub-pixel materials, etc. will require more paths (samples) and a pixel imaging a uniform, low variability surface will require fewer paths( samples).

The adaptive pixel sampling algorithm has three parameters: (1) the minimum number of paths, the (2) the maximum number of paths and (3) the threshold for determining if the solution has converged.

  • The minimum number of samples specifies the minimum number of paths used in a pixel.

    • "Why wouldn’t I want this to be 1?"

      • Because the Monte-Carlo integral of a complex function with 1 sample will be poor. This will result in an incredibly very noisy image.

    • "What is the default?"

      • The default value is currently 20, which produces images that are decent quality as long as the illumination is generally high.

    • "Why would I increase the minimum number of samples?"

      • With only 20 samples, you might still get noise in the shadows because 20 wasn’t enough to detect the large dynamic range within a complex illumination environment (for example, shadows under a tree canopy).

  • The maximum number of samples specifies the maximum number of paths used in a pixel.

    • In general, this is the primary "quality" knob.

      • Turn it up to get more samples and drive down noise in the Monte-Carlo integration.

    • "What is the default?"

      • The default value is currently 100, which produces images that are decent quality as long as the illumination is generally high.

    • "Why would I increase the maximum number of samples?"

      • In general, it helps areas that are noisy. For example, shadow regions with complex illumination.

  • The threshold at which additional paths will no longer be add to a pixel.

    • Computed using the average radiance at the reference wavelength

    • This value is in units of radiance - it is not a fraction!

      • This value should be selected in terms of radiometry and not to make something pleasing to the eye.

      • Long term we expect the sensor model to define this value in terms of a minimal detectable radiance.

  • All three parameters are related:

    • At one extreme, if the threshold is very low (for example, 0) then each pixel will use the maximum number of paths.

    • At the other extreme, if the threshold is very high (for example, 1e100) then each pixel will use the minimum number of paths.

The forth key parameter is the number of nodes (bounces) along each path. This is controlled via the --max_nodes option. The default number of nodes is 4, but the user can increase it for highly scattering scenarios.

A summary of suggested convergence and node parameter configurations is listed in the table below.

Table 1. A table of suggested convergence and node count values for different simulations.
Mode Min Paths/Pixel Max Paths/Pixel Radiance Threshold Max Nodes

Default

20

100

1.0e-06

4

Better

30

500

1.0e-06

5

Even better

50

1000

1.0e-06

5

MWIR

20

100

1.0e-08

4

LWIR

20

100

1.0e-08

4

Night

30

500

1.0e-12

5

Lidar

50

100

1.0e-12

4

Preview

3

3

0.0

1
Tip The radiance levels are lower in the MWIR and LWIR regions, so the radiance threshold (which is absolute) needs to be lowered.
Tip At night the radiance values are much lower, so the radiance threshold needs to be decreased in order to match the magnitude range.
Tip The lidar instrument automatically adjusts the minimum paths/pixel and radiance threshold to at least match the values shown here. If the user specifies a minimum paths/pixel of less than 50 or a radiance threshold of greater than 1.0e-12, then the respective values will be modified automatically.
Tip The user can emulate the DIRSIG4 "preview mode" by supplying the -preview option to DIRSIG5. However unlike DIRSIG4, which would skip radiance calculations in this mode, the DIRSIG5 simulation will produce a low-fidelity radiance solution.

Controlling Multi-Threading

The software will detect the number of virtual cores (Intel Hyper-threading efficiently allows more than 1 thread to run on each physical core) and will automatically try to assign a thread to each. That will yield the highest utilization of the host computer’s resources, but in a shared computing environment this might not be desired. Hence, the user can specify the number of threads to start via the --threads option. For example, to use only 4 threads:

$ dirsig5 --threads=4 example.sim

The user can also modify the thread "block size" from the command-line using the --thread_blocksize option. This option sets the number of problems assigned to a thread at once.

$ dirsig5 --thread_blocksize=4 example.sim

Smaller values lead to finer-grained thread scheduling, which can reduce thead downtime but with an increased cost of thread initialization. Conversely, larger values lead to coarser thread scheduling, potentially leaving some threads without work on occasion, but reducing the cost of thread initialization. Typical values for the block size should be between 1 and 16. The default is 8.

Warning This is an advanced feature. Do not change this value unless you know what you are doing. This option is typically employed when the size of the focal plane is small (e.g, a lidar that has only a handful of detectors).

Controlling Output Verbosity

The DIRSIG5 model leverages the spdlog library to offer multiple levels of verbosity in terms of console output. The default level is referred to as "info", which outputs standard information about the progress of the simulation as it executes. This default can be explitly requested by setting the --log_level option to info (as shown below):

$ dirsig5 --log_level=info example.jsim
DIRSIG5
Version: 2022.08 (3cf99f2) built on Feb 23 2022 16:20:53
Copyright 2015-2022 Rochester Institute of Technology

Hostname: 'dirsig.local'
Date/Time: Wed Feb 23 16:31:37 2022

Plugin status:
    Found 48 installed plugin(s)

If you are encountering a problem with DIRSIG, it might be useful to change the log level to the "debug" state. This will generate significantly more output by including a messages about various intermediate steps, updates, etc. in the model. To enable this output set the --log_level option to debug (as shown below):

$ dirsig5 --log_level=debug example.jsim
DIRSIG5
Version: 2022.08 (3cf99f2) built on Feb 23 2022 16:20:53
Copyright 2015-2022 Rochester Institute of Technology

Hostname: 'dirsig.local'
Date/Time: Wed Feb 23 16:32:10 2022

[debug] Command-line invocation:
[debug] dirsig5 --log_level=debug example.json
[debug] DIRSIG Config:
[debug]   Simulation filename: /Users/dirsig/example.json
[debug]   Runtime settings:
[debug]     Max threads: 10
[debug]     Default threads: 10
[debug]   Max nodes: 4
[debug]   Using earth core
[debug]   Log level: debug
[debug] Loading plugins from directories
[debug] Searching user's default plugin directory
[debug] '/Users/dirsig/Documents/DIRSIG/plugins' -> found 0 plugin(s)
[debug] Searching installation plugin directory
[debug] '/opt/local/dirsig/plugins' -> found 48 plugin(s)

Note that all of the "debug" messages will be have lines that start with [D] and in compatible shells, the text will be cyan. It should also be noted that debug messages are sent to the standard error stream rather than standard output.

Regardless of the log level, error messages produced by DIRSIG will be sent to standard error stream and each line will begin with [E] and the text will be red on compatible shells.

[error] Missing simulation filename!

Generating JSON Logs

DIRSIG5 has two options that produce JSON formatted files that capture details about the simulation and store them in an easily parsed JSON file(s).

Requesting the Run Log

The "run" log contains information about the scenes and plugins loaded for the simulation:

  • The list of scenes and their respective info including,

    • The HDF filename,

    • The bounding box,

    • The geolocations and

    • The MD5 hash for the compilation.

  • The list of plugins and their respective inputs. In addition, each plugin can write their own info to the log. For example, the BasicPlatform plugin (the most commonly used sensor plugin) will write:

    • For each instrument

      • The focal length

      • For each focal plane:

        • The array geometry (pixel sizes, offsets, counts, etc.),2s

        • The spectral information (bandpass min, max and sampling interval, the channel list, etc.),

        • The temporal information (clock rate, integration time, etc.)

        • The primary image (filename, radiometric units, etc.)

        • The truth image (filename, truth requests, etc.).

This log file can be created via the --run_info option, which is supplied the name of the JSON file to be created:

$ dirsig5 --run_info=run_log.json example.jsim

Requesting the Info Log

The "info" log focuses on the series of captures that will be performed by the simulation. Similar to the "run" log, each sensor plugin is responsible for the information in each capture summary. In the case of the BasicPlatform plugin (the most commonly used sensor plugin), it will write the following for each capture:

  • The capture time

  • A description (typically the name of the focal plane and instrument)

  • Exo-atmospheric source summary (Sun and Moon)

  • The platform location and orientation

  • The platform-relative mount location and orientation

  • The X/Y field-of-view of the focal plane being capture

  • The image lat/lon/alt corner points

  • The primary image filename

  • The truth image filename

This log file can be created via the --log_info option, which is supplied the name of the JSON file to be created:

$ dirsig5 --log_info=info_log.json example.jsim
Note If the --run_log option is not also supplied, it will be included in the "info" log file as well.

Performing a "Dry Run"

The "dry run" option in DIRSIG will load the simulation, the scenes, the requested plugins, etc. and perform most tasks except for executing all the captures defined by the simulation. This option is useful when combined with the log options described previously as it provides a way to generate those logs without actually waiting for the radiometric calculations to run. This option must be combined with the --log_info option.

$ dirsig5 --dry_run --log_info=example_log.json example.jsim

This feature is useful for gathering information about the captures to be performed, including the names of files that will be created.

Output Folder and File Options

DIRSIG5 supports the same DIRSIG4 options to specify an optional folder into which output files will be placed and/or prefix string for output files.

Specify the Output Folder

This option allows the user to specify the folder (directory) that all output files should be written to:

$ dirsig5 --output_folder=/Users/dirsig/output/sim1 ...

For example, if a focal plane routes data to an file with the name band1.img, the file will be /Users/dirsig/output/sim1/band1.img

Specify an Output File Prefix

This option allows the user to specify a prefix for all output files that are written to:

$ dirsig5 --output_prefix=run1_ ...

For example, when combined with the output folder option above, the same band1.img file will be /Users/dirsig/output/sim1/run1_band1.img

Modifying the Event Schedule

Normally the sensor plugins will submit 1 or more sets of captures (referred to as "tasks") to define the scope of a simulation. In some situations, the user will not want to run all of those captures. To support that need, there are a series of options that allow the user to modify the capture event schedule generated by the sensor plugins.

To print the capture event schedule, the user can specify the --dry_run option:

Running DIRSIG in the "dry run" mode.
$ dirsig5 --dry_run example.sim

Each capture event in the queue includes the index of the sensor plugin, the index of the task and the index of the capture within the task. This index triplet can be used specify any capture in the event queue. Indexes start at 0.

Running individual captures

To run a specific, individual capture in the simulation, the --capture option can be used. In the following example, capture 11 within task 7 for sensor plugin 0 will be run.

Running DIRSIG for the 12th capture, in the 8th task of the 1st sensor plugin.
$ dirsig5 --capture=0:7:11 example.sim

Running individual tasks

To run all the captures in a specific task, the --task option can be used. In the following example, all the captures within task 7 for sensor plugin 0 will be run.

Running DIRSIG for all the captures in the 8th task of the 1st sensor plugin.
$ dirsig5 --task=0:7 example.sim

Running multiple tasks and/or captures

The --capture and --task options can be combined and/or supplied multiple times to specify a set of captures to be executed from the baseline queue. The following example runs all the captures in task index 1, then all the captures in task index 2, then capture index 1 in task index 3 and finally capture index 2 in task index 3:

Running DIRSIG for individual tasks and captures.
$ dirsig5 --task=0:1 --task=0:2 --capture=0:3:1 --capture=0:3:2 example.sim

Running part of a task

To start at a given capture and complete all the remaining captures in the event queue the --start_capture option can be used. In the following example, the event queue will start at capture 11 within task 7 for sensor plugin 0 and then continue through the remaining capture events in the queue.

Starting DIRSIG at the 12th capture in the 8th task for the 1st sensor plugin.
$ dirsig5 --start_capture=0:7:11 example.sim

The --start_capture option can also be combined with the --end_capture option to select a window of captures:

Starting DIRSIG from the 12th to the 21st capture in the 2nd task for the 1st sensor plugin.
$ dirsig5 --start_capture=0:1:11 --end_capture=0:1:20 example.sim

Completing a partially completed simulation

The --capture and --start_capture options also allow the user to optionally specify the starting problem within the capture (or within first capture in the case of the starting capture option). This option is useful in the situation where a capture was partially completed by a previous simulation. Consider a sensor that collects a 1280x720 frame in each capture. That translates into 1280 x 720 = 921,600 problems per capture. If the simulation only completed the first 300 lines of the capture, that means 1280 x 300 = 384,000 problems were completed. The user can finish this capture by specifying the optional starting problem index by adding a 4th field to the index triplet:

Completing the 12th capture in the 8th task for the 1st sensor plugin.
$ dirsig5 --capture=0:7:11:384000 example.sim

or, finish that capture and all the captures after it:

Completing the 12th capture in the 8th task for the 1st sensor plugin and all remaining tasks and captures.
$ dirsig5 --start_capture=0:7:11:384000 example.sim
Note If the user supplies the starting problem index and the output for the previous problems have not been generated, most sensor plugins will pad/fill the file with 0s and then start writing the data.
Important Due to how captures are computed across multiple nodes, the problem index (4th field) option is not supported by the MPI-enabled version of DIRSIG5.

Display 3rd Party Licenses

The user can print the 3rd party licenses using either the --license or -licenses option:

$ dirsig5 --license

or

$ dirsig5 --licenses

Using the Parallel/MPI version

Details about using the MPI-enabled version of DIRSIG is described in a supplementary manual.

Plugins

The list of the documented plugins employed by DIRSIG5 can be found in the Plugins Manual.

Implementation Notes

A collection of documentation pages containing some behind-the-scenes implementation details for nerds.