Keywords: chips, machine learning

Summary

This is an advanced demonstration of using the ChipMaker combo plugin. In order to facilitate robust training of machine learning algorithms, we want a large number of images featuring the content of interest. These image "chips" should span a wide range of acquisition parameters. This plugin was created to streamline the generation of image chips with "labels" by automating the sampling of a user-defined parameter space in a single simulation rather than relying on an infrastructure of external scripts to iterate through the parameter space.

Important
 This is a DIRSIG5 specific feature and cannot be used in DIRSIG4.

Details

In order to facilitate robust training, we want to generate a large number of image chips across a wide range of acquisition parameters. Those include:

  • Different targets and/or variants of targets

  • Different backgrounds (the context within which the target appears)

  • Different ground sampling distances (GSDs)

  • Different sensor view angles (zenith and azimuth)

  • Different illumination angles (zenith and azimuth)

Historically this has been accomplished using external scripting with a conventional DIRSIG simulation. The primary goal of this plugin is to make it easy to configure all the degrees of freedom in one location and have the plugin manage the creation of the images.

Important Files

This section highlights key files important to the simulation.

Scene

The scene was developed specifically for this demo. Although the scene is very simple, it illustrates techniques that can be used to develop other scenes appropriate for use in this algorithm training applications. The goal of the scene construction approach at this time is to create a large number of contexts for each target of interest. In this case, the context primarily refers to the background. To create a robust training set for a given target, we want images of that target against a large number of backgrounds. There are currently two ways to accomplish this with the current DIRSIG workflow:

  • Create a scene with a divergse background and create a large number of instances of the target within that scene. Each instance of the target will be in a unique location and, hence, have a unique background. In this demo, the helicopters use this approach.

  • Create a scene with a diverse background and use motion of a small set of target instances that are in motion. When the targets are imaged at different times, there will be a unique background around the object. In this demo, the bomber plane uses this approach.

The Background

The background for the scene is defined as a flat terrain with a material (class) map defining asphalt, concrete, grass and dirt with texture maps. This object is defined in a bundle (see geometry/bundles/background).

The KA-27 Helicopters

The KA-27 helicopter is defined in a set of bundles. There are 6 total variants of the KA-27: 3 materials variants (Indian, Korean and Russian) and 2 state variants (idle and active). The material variants use a unique set of materials for each vehicle (different reflectance maps featuring nation specific paint schemes and markings). The 2 state variants define if the rotors on the vehicles are in motion or not. This results in 6 permutations of the 3 materials and 2 states. These are captured in the 6 GLIST files within the KA-27 bundle folder:

  • geometry/bundles/ka27/ka27_indian_active.glist

  • geometry/bundles/ka27/ka27_indian_idle.glist

  • geometry/bundles/ka27/ka27_korean_active.glist

  • geometry/bundles/ka27/ka27_korean_idle.glist

  • geometry/bundles/ka27/ka27_russian_active.glist

  • geometry/bundles/ka27/ka27_russian_idle.glist

In order to select these objects in the ChipMaker, the target objects have the tags defined at the top-level geometry list in the bundle GLIST files. The example below is from the ka27_russian_idle.glist file showing the tags defined:

<geometrylist tags="ka27,choppers,russian,idle,aircraft">
  <object search_paths="local">
    <localmaterials>materials/chopper.mat</localmaterials>
    <basegeometry>
      <obj>
        <filename>ka27.obj</filename>
        <assign id="ka27_body_russian">body</assign>
      </obj>
    </basegeometry>
    <staticinstance/>
  </object>
</geometrylist>

The "Badger" Bomber Plane

The "Badger" bomber is defined in a bundle (see geometry/bundles/badger). There is only a single instance of the bomber in the scene, which is defined in the geometry/planes.glist file.

In order to select these objects in the ChipMaker, the target objects have the tags defined at the top-level geometry list in the bundle GLIST files. The example below is from the badger.glist file showing the tags defined:

<geometrylist tags="active,badger,plane,aircraft">
  <object enabled="true" search_paths="local">
    <localmaterials>materials/badger.mat</localmaterials>
    <basegeometry>
      <obj>
        <filename>badget_edited.obj</filename>
      </obj>
    </basegeometry>
    ...
  </object>
</geometrylist>

The single instance of the bomber is dynamic, using the DeltaMotion model to define a take-off of the plane over the runway in the scene. The total takeoff sequence described in the motion file (see geometry/motion/PlaneTakeOff.mov) spans from 0 - 150 seconds. This time window will be used to capture the plane at various states in the take off sequence. Note that we have added the tag "flying" to the dynamic instance. This tag will be added to the tags for this instance of the badger

<geometrylist>
  <object enabled="true">
    <basegeometry>
      <glist>
        <filename>bundles/badger/badger.glist</filename>
      </glist>
    </basegeometry>
    <dynamicinstance tags="flying">
      <motion type="delta">
        <filename>motion/PlaneTakeOff.mov</filename>
      </motion>
    </dynamicinstance>
  </object>
</geometrylist>
images/demo.png
Figure 1. A rendering of the complete scene.

The key feature of this scene is that it utilizes the "tagging" system introduced for DIRSIG5 to add shared and unique "labels" to each of the 21 objects in the scene. These tags will be used in the simulation to select which objects will be sampled in the output image set.

Geometry Tags

A more detailed explanation of the tagging system and how tags are inherited is described in the ChipMaker1 demo. Below is a list of the tags in the scene and what targets they map to:

aircraft

Maps to all aircraft (helicopters and planes) in the scene.

choppers

Maps to all the helicopters in the scene.

planes

Maps to all the planes in the scene.

ka27

Maps to all the KA-27 helicopters in the scene.

badger

Maps to all the "Badger" planes in the scene.

indian

Maps to all the Indian aircraft in the scene (in this case, the idle and active KA-27 helicopters).

korean

Maps to all the Korean aircraft in the scene (in this case, the idle and active KA-27 helicopters).

russian

Maps to all the Russian aircraft in the scene (in this case, the idle and active KA-27 helicopters).

idle

Maps to all the idle aircraft in the scene (in this case, the idle KA-27 helicopters).

active

Maps to all the active aircraft in the scene (in this case, the active KA-27 helicopters and the Badger).

flying

Maps to all the active aircraft in the scene that are airborne (in this case, the only the Badger is ever airborne).

Configuration File

The input file for the plugin is a JSON formatted file. Refer to the ChipMaker plugin manual for specific details about the format and options of this file. The input file used in this demo is included below with a brief discussion.

{
    "camera" : {
        "image_size" : {
            "x" : 180,
            "y" : 180
        },
        "gsd_range" : {
            "minimum" : 0.25,
            "maximum" : 0.35
        },
        "channellist" : [
            {
                "name" : "Pan",
                "minimum" : 0.4,
                "maximum" : 0.8
            }
        ],
        "psf" : {
            "image" : "gaussian.png",
            "scale" : 10
        },
        "readout" : {
            "integration_time" : 0.010
        },
        "image_filename" : {
            "basename" : "chip",
            "extension" : "img"
        },
        "truth" : [
            "scene_x", "scene_y", "scene_z", "geometry_index"
        ]
    },
    "time_range" : {
        "minimum" : 100,
        "maximum" : 120
    },
    "view" : {
        "zenith_range" : {
            "minimum" : 10,
            "maximum" : 30
        },
        "azimuth_range" : {
            "minimum" : 0,
            "maximum" : 360
        }
    },
    "source" : {
        "zenith_range" : {
            "minimum" : 10,
            "maximum" : 40
        },
        "azimuth_range" : {
            "minimum" : 80,
            "maximum" : 280
        }
    },
    "setup" : {
        "target_tags" : [ "plane" ],
        "count" : 25,
        "options" : ["hide_others"],
        "report_filename" : "plane_labels.txt"
    }
}

The output images will all be 180 x 180 with the GSD ranging from 0.25 to 0.35 meters with a PAN channel. The time range is set to 100 - 120 seconds so that we capture the plane during the touchdown of the landing sequence. The integration time is set to 10 ms (0.01 seconds) to capture any motion blur of the plane. The output images will be named chip0.img, chip1.img, etc. In addition to the single radiance channel, the output image files will include 4 "truth" channels that include the Scene XYZ for each pixel and the "geometric index" (which can use used with the geometry_report.txt file to resolve which instance is captured in each chip). The camera can look at the targets from 10 to 30 degrees off-nadir and anywhere in azimuth. Similarly, the source (sun) can illuminate the scene from 10 - 40 degrees off nadir and from 80 - 280 degrees azimuth. The output image set will consist of 25 images and the target set will be anything that is. The (optional) label report file will be saved as plane_labels.txt.

Simulation File

A chip generation simulation is a special type of simulation that requires a DIRSIG5 JSON simulation file. Below is the contents of the chips.jsim file:

[{
    "scene_list" : [
        { "inputs" : "./demo.scene" }
    ],
    "plugin_list" : [
        {
            "name" : "ChipMaker",
            "inputs" : {
                "input_filename" : "./chips.json"
            }
        }
    ]
}]
Note
 This input simulation file cannot be loaded into the graphical simulation editor.

Setup

Generating plane images

To run the simulation, start the DIRSIG5 program on the command-line with the plane_chips.jsim file:

$ dirsig5 plane_chips.jsim

As the model runs, a short message will appear regarding the number of instances matched by the requested tags. In this case, it correctly found 2 targets (one "parked" and one "flying"). Since we are generating 25 images, we will image several of these targets many times. As each image chip is generated, a small output message will be written to the output with the random parameters used for that chip. The "target index" indicates which of the 2 targets was selected for that image.

DIRSIG5
Release: 5.0.10 (r2582M)
Build Date: Dec 19 2018 17:23:47
Copyright 2015-2018 Rochester Institute of Technology

Checking Scenes:
Opened scene: "./demo.scene.hdf"
Bounding box (lat/lon/alt):
  minimum: -309.6 -478.4 -0.001
  maximum: 309.6 478.4 10.4614

Loading Plugins:
Setting up the 'ChipMaker' plugin (input = ./plane_chips.json)
Requesting instances with tag(s):
    plane
Primary scene location: 43 +N, 78 +W, 0 [m]

Running in multi-threaded mode with 8 threads

Initializing the scheduler ... done.

Running the simulation

Initializing the 'ChipMaker' plugin:
    Random seed = 658
    Using convergence parameters = 20, 100, 1.000e-06
    Using a maximum of 4 nodes per path
    Checking target tag requests:
        'plane': found 2 instance(s)
        2 total target(s) found
    Image chip count = 25
        Image size = 180 x 180
        Image basename = chip
    Truth requests:
        X Hit Coordinate [m]
        Y Hit Coordinate [m]
        Z Hit Coordinate [m]
        Geometry Index
    Pre-computing chip parameters ... done.

Starting chip #0
    Target index = 1
    Tags = flying, badger, plane, aircraft, active
    Location = 2.629e+02,-2.696e+02,4.697e+01
    Offset = 0.000e+00,0.000e+00,0.000e+00
    Time = 1.131e+02
    GSD = 3.285e-01
    View angles = 2.706e+01, 3.124e+02
    Source angles = 1.672e+01, 2.225e+02

Results

Plane images

images/plane_chips.png
Figure 2. The images for the "planes" selection set.

The other key output is the label report, which contains a comma-separated row of parameters for each image chip. Below is the plane_labels.txt file:

# DIRSIG5 'ChipMaker' label report
# Generated on Fri Mar 15 11:53:10 2019
#
# Input parameters:
#     Random seed = 650
#     Image size = 180 x 180
#     Time range = [100:120]
#     GSD range = [0.25:0.35]
#     View zenith range = [10:30]
#     View azimuth range = [0:360]
#     View offset range = [0:0]
#     Source zenith range = [10:40]
#     Source azimuth range = [80:280]
#
# Fields:
# targetIndex, X, Y, Z, time, GSD, viewZen, viewAzi, sourceZen, sourceAzi, tags
0,0,-90,375,0,117.599,0.278749,22.4733,232.082,34.8964,198.119,"parked:badger:plane:aircraft:active"
1,1,262.892,-336.043,57.0948,119.118,0.324486,13.3063,304.117,25.2511,131.569,"flying:badger:plane:aircraft:active"
2,0,-90,375,0,118.016,0.263938,28.2886,277.251,31.7074,143.393,"parked:badger:plane:aircraft:active"
3,0,-90,375,0,109.299,0.328899,25.3204,224.652,24.6061,116.35,"parked:badger:plane:aircraft:active"
4,0,-90,375,0,104.774,0.330986,29.432,70.0678,30.8078,81.5123,"parked:badger:plane:aircraft:active"
5,1,262.892,-282.857,49.2271,114.327,0.254662,12.535,179.402,19.119,130.689,"flying:badger:plane:aircraft:active"
6,1,262.892,-240.515,41.6136,110.494,0.309903,24.9769,0.106731,33.9668,146.441,"flying:badger:plane:aircraft:active"
7,0,-90,375,0,117.613,0.278681,13.9235,49.2909,39.3508,145.493,"parked:badger:plane:aircraft:active"
8,0,-90,375,0,107.964,0.328685,19.3926,166.087,32.2435,126.778,"parked:badger:plane:aircraft:active"
9,1,262.892,-228.804,39.294,109.43,0.320495,10.9795,256.394,11.2589,83.732,"flying:badger:plane:aircraft:active"
10,1,262.892,-178.982,28.3761,104.886,0.263345,21.9115,153.293,13.336,265.872,"flying:badger:plane:aircraft:active"
11,1,262.892,-339.21,57.5044,119.402,0.304839,17.2521,339.849,35.7509,184.613,"flying:badger:plane:aircraft:active"
12,0,-90,375,0,103.552,0.311723,27.763,223.276,35.1643,82.0975,"parked:badger:plane:aircraft:active"
13,0,-90,375,0,116.552,0.295408,25.7081,341.612,11.244,129.288,"parked:badger:plane:aircraft:active"
14,1,262.892,-336.045,57.0951,119.118,0.333699,11.714,30.928,27.4111,112.683,"flying:badger:plane:aircraft:active"
15,0,-90,375,0,115.016,0.281598,17.1845,282.722,37.0417,223.038,"parked:badger:plane:aircraft:active"
16,0,-90,375,0,105.576,0.304214,13.111,208.036,23.4894,97.3484,"parked:badger:plane:aircraft:active"
17,0,-90,375,0,113.957,0.301116,22.6629,78.7787,13.8918,128.61,"parked:badger:plane:aircraft:active"
18,1,262.892,-156.986,23.0085,102.868,0.313683,24.1875,36.2972,39.6899,100.067,"flying:badger:plane:aircraft:active"
19,1,262.892,-211.173,35.6254,107.826,0.253148,17.4392,339.537,12.1514,271.57,"flying:badger:plane:aircraft:active"
20,0,-90,375,0,106.708,0.346194,12.0769,316.948,10.1301,157.73,"parked:badger:plane:aircraft:active"
21,1,262.892,-287.496,49.9881,114.746,0.261275,26.4576,289.819,13.5948,184.798,"flying:badger:plane:aircraft:active"
22,1,262.892,-287.394,49.9715,114.737,0.250107,24.4218,55.9667,12.2976,225.202,"flying:badger:plane:aircraft:active"
23,0,-90,375,0,101.357,0.25031,23.78,104.247,35.1732,273.617,"parked:badger:plane:aircraft:active"
24,1,262.892,-257.061,44.7325,111.994,0.311358,13.4397,207.051,21.5705,146.608,"flying:badger:plane:aircraft:active"

Chopper images

images/chopper_chips.png
Figure 3. The images for the "choppers" selection set.