Keywords: chips, machine learning

Summary

This is demonstration of the ChipMaker comboi plugin with an alternative scene model approach that leverages a collection of smaller area background scenes (referred to as scene-lets) and a collection of target scenes that are combined on-the-fly to facilitate greater variation of targets and backgrounds.

Related Materials

The following demos, manuals and tutorials can provide additional information about the topics at the focus of this demo:

  • Related Demos

  • Related Manuals

  • Related Tutorials

    • None.

Details

The primary difference in this usage of ChipMaker compared to the one outlined in the ChipMaker2 demo is how the target and background scene is represented in each image chip. The approach described here leverages a collection of smaller background scenes (without any target present) and target-only scenes. The simulation then randomly selects a target and background combination for each image chip. The benefits of this approach are the following:

  • The user doesn’t need to worry about traditional, large-area scene construction proceedures. Instead, a library of small-area scene-lets can be created and used with any target.

  • The user doesn’t need to worry about placing a large number targets in a single large-area background. Instead, targets are developed as stand-alone scenes and can be combined with backgrounds at run time.

Important Files

This section highlights key files important to the simulation.

The Background Scene Files

To keep the size of the demo small, this example uses only 2 background scenes. But the concept here is intended to scale to 10s or 100s of small background scenes. The 2 example scenes here are flat and relay on traditional DIRSIG material variation methods to create some complexity. However, the background scenes do not need to be this simple and can (and should) normally feature 3D complexity when possible. The two background scenes are:

  • The grass_field.scene, which is a flat plate with a textured grass material assigned to it. The geometry setup for this scene is the geometry/grass_field.glist file.

  • The parking_lot.scene, which is a flat plate with a textured asphalt and painted parking lot lines. This is essentially the same background used in the Parking1 demo. The geometry setup for this scene is the geometry/parking_lot.glist file.

Both of the scenes leverage the same material database (see materials/demo.mat) but that is by no means a requirement. Background scene-lets can be created using any valid scene construction approach including completely separate scene directory hierarchies, bundled objects in a single scene directory hierarchy, etc.

Important
 It is assumed that background scenes have the Scene ENU origin in the middle of the scene.

The Target Scene Files

There are 4 target vehicles that are available in the simulation. A target scene contains nothing but the target so that it can be easily combined with a background scene on-the-fly. The target vehicles in this scene were taken from the Parking1 demo. The 4 target scenes are:

  • The generic_pickup.scene contains a generic pickup truck.

  • The infiniti_g35.scene contains an Infinity G35 sedan.

  • The nissan_fairlady.scene contains a Nissan Fairlady sedan.

  • The toyota_corolla.scene contains a Toyota Corolla sedan.

Important
 It is assumed that background scenes have the Scene ENU origin at the "anchor point" (usually the XY midpoint and Z minimum).

The JSIM File

The Scene List

The key to this approach is captured in scene_list array of the JSIM file. The scene list will pull all of the target and background scenes into the simulation and the ChipMaker plugin will created target and background combinations for each image chip using the following approach:

  • Each scene is tagged as either a target or background scene. An additional unique tag must also be provided for each scene.

  • At runtime, the ChipMaker plugin creates a list of background scenes and a list target scenes by requesting lists from the radiometry core using the respective tag names.

  • For each image chip, the ChipMaker plugin picks a target scene and a background scene from the respective lists. It then tells the radiometry core to disable all the scenes except for those two scenes. What remains is the target and background scene combination for that image chip.

Important
 All the target and background scenes are simply overlaid on top of each other. The target scene is not insert onto the background scene. Therefore, the recommended target and background scene coordinate assumptions outlined above must be followed to avoid targets getting emedded in background scenes.

The excerpt of the demo.jsim file below shows the list of target and background scene files used in this simulation. Note the use of the tags to denote which scenes are in the target list and which are in the background list:

The scene list contains target and background scenes, each tagged approptiately.
    "scene_list" : [
        {
            "inputs" : "./grass_field.scene",
            "tags" : "background,grass_field"
        },
        {
            "inputs" : "./parking_lot.scene",
            "tags" : "background,parking_lot"
        },
        {
            "inputs" : "./generic_pickup.scene",
            "tags" : "target,generic_pickup"
        },
        {
            "inputs" : "./infiniti_g35.scene",
            "tags" : "target,infiniti_g35"
        },
        {
            "inputs" : "./nissan_fairlady.scene",
            "tags" : "target,nissan_fairlady"
        },
        {
            "inputs" : "./toyota_corolla.scene",
            "tags" : "target,toyota_corolla"
        }
    ],

The ChipMaker Plugin

The primary difference in the ChipMaker plugin setup is that rather than use the target_tags mechanism in the setup section of the JSON configuration to select targets in a single scene, the plugin will use the scene selection mechansim when the following setup configurations are met:

  • The target_tags variable is not set, and

  • The target_range section is set.

The target_range defines a axis aligned box in 3D space that will be used to compute a random look point for each image.

The setup section doesn’t include the "target tags" but does include the "target range".
                "setup" : {
                    "count" : 20,
                    "random_seed" : 1,
                    "target_range" : {
                         "minimum" : [ -0.5, -0.5, 0 ],
                         "maximum" : [  0.5,  0.5, 0 ]
                    },
                    "options" : ["hide_others","make_meta"]
                }

Again, the assumptions introduced above were that the background scenes would be centered around 0,0,0 in the scene ENU coordinate system and that the target scenes would be anchored at 0,0,0 in the scene ENU coordinate system. Hence, a target_range that looks at 0,0,0 would produce images with the target in the center. In this case, we opened up the range in X and Y so that the target is not in the center of every image.

Simulations and Results

Run the demo.jsim file using DIRSIG5:

$ dirsig5 demo.jsim

The ChipMaker setup included 4 radiometric bands (red, green, blue and pan) plus 1 truth band (scene index), so the output image chips have 5 bands total. To create RGB images, we can use the image_tool and specify bands 0, 1 and 2 (R, G and B, respectively).

$ image_tool convert --autoscale=gamma --format=png --bands=0,1,2 chip*.img
Important
 The above syntax only works on Linux and MacOS, where wildcards are expanded by the shell. The image_tool manual explains a method for how to use wildcards on Windows.
chips rgb
Figure 1. An animated GIF of the 20 RGB chips produced by this simulation.

To create PAN images, we can specify the pan band (band index = 3) for the RGB channels:

$ image_tool convert --autoscale=gamma --format=png --bands=3,3,3 chip*.img
chips pan
Figure 2. An animated GIF of the 20 PAN chips produced by this simulation.

To create target mask images, we can specify the scene index truth band (band index = 4) for the RGB channels:

$ image_tool convert --autoscale=gamma --format=png --bands=4,4,4 chip*.img
chips mask
Figure 3. An animated GIF of the 20 target mask chips produced by this simulation.

It should also be noted that the make_meta option was supplied to the options array in the setup section, which means each chip will have a .meta JSON formatted file generated for it that contains information about that chip:

Example JSON meta-data file for an image chip.
{
    "image_filename" : "chip10.img",
    "image_size" : [320,240],
    "target_index" : 0,
    "time" : 101.015,
    "target_location" : [0.491123,0.11632,0],
    "gsd" : 0.05,
    "view_zenith" : 0.293853,
    "view_azimuth" : 4.9167,
    "view_distance" : 1000,
    "source_zenith" : 0.378466,
    "source_azimuth" : 2.6426,
    "atm_conditions" : "",
    "scene_tags" : ["toyota_corolla", "parking_lot"],
    "keypoints" : [
    ]
}

Note that in addition to the imaging parameters for the chip (e.g., the GSD, view geometry, source geometry, etc.) the scene_tags variable includes the tags for the target and background scenes used in that image chip.