This tutorial explains the workflow to extract BRDF data from the Non-Conventional Exploitation Factors (NEF) Database so that it can be captured as a SQT and DHR file and used in a DIRSIG5 simulation.

Note This tutorial has only been tested on a Windows computer and is intended to serve as a guideline since various edits are expected depending on the host operating system, installation details, etc.

Overview

DIRSIG4 and NEF 11 and Earlier

The NEF database contains bi-directional reflectance function (BRDF) and bi-directional transmittance function (BTDF) data for a large number of materials. Most of these material descriptions were created by fitting lab BRDF/BTDF measurements to a variety of parametric models, including the standard Hapke model and the "modified" Beard-Maxwell (MBM) model. Previous DIRSIG4 integrations with the NEF involved duplicating the code for the Hapke model and the NEF’s unique MBM model as native BRDF models in DIRSIG and importing the corresponding parameters from the NEF file-based material descriptions.

When NEF version 12 was released, it reflected a major software overhaul effort, which included the migration from a file-based database to an SQL database. The updated NEF database and utility software now included a Java-based application programming interface (API) — the same API used by the graphical user interface (GUI) provided with software. The API included tools to compute the BRDF for a defined source and view geometry. However, using the API was too slow for DIRSIG4 to leverage for run-time calculations. Although the API could disclose which BRDF model is used for a given material, it did not provide direct access to the BRDF model parameters which could be used by DIRSIG4’s existing implementations of Hapke and Modified Beard-Maxwell. Without a computationally efficient way to perform BRDF calculations at run-time, support for the NEF database in DIRSIG4 came to an end.

DIRSIG5 and Spherical Quad Trees

When DIRSIG5 was created, one of the key design decisions was to avoid developing, validating and maintaining dozens of BRDF model implementations in the new code base. In addition, we desired a workflow that could ingest measured BRDF data more efficiently (potentially without employing a parametric model for fitting). And finally, DIRSIG5 would need more than just the BRDF "evaluation" calculation (e.g. the BRDF for a given source and view geometry), it would need efficient ways to perform two additional tasks:

  1. Efficiently integrate the BRDF into a directional hemispherical reflectance (DHR), and

  2. Efficiently importance sample the BRDF (e.g. treat the BRDF as a probability function and pseudo-randomly draw reflected vectors).

These requirements resulted in the development of Spherical Quad Trees (SQTs) in DIRSIG5. The SQT mechanism solved all of these requirements by utilizing a data-driven representation of hemispherical or spherical data that could be quickly evaluated, integrated and sampled. A toolbox of utility programs were created to import data into the SQT formatted files, and extract data from SQT formatted files for data visualization and evaluation. In DIRSIG5, all parameterized BRDF models are angularly sampled and then imported into this data-centric SQT format for use at run-time. A key tool in the SQT data workflow is the raw2sqt tool, which imports angularly sampled spectral data from a documented, ASCII/Text file and optimizes into a binary SQT file. Since the Java-based NEF API provided a way to sample any BRDF model in the database and write it into one of these ASCII/Text intermediate files, an integration mechanism was now possible once again.

DIRSIG5 Integration Approach

The nef2raw tool used in this tutorial uses the NEF’s Java-based API to extract two components of the spectral-angular material description from a material in the NEF database:

  1. The BRDF is extracted at a discrete set of wavelengths, and

  2. The spectral DHR is extracted at the "best" spectral resolution supported by that material.

Although the API supports evaluating the BRDF at any wavelength, the reality is that underlying description tends to have model parameters at no more than 5 wavelengths: the UV, VIS, NIR, MWIR and LWIR. This reflects that fact that although the DHR can vary rapidly by wavelength, the shape of the BRDF generally does not. Extracting the full BRDF at 1000’s of wavelengths would result in a lot of data to be stored. Hence, the approach in the nef2raw tool is to extract the BRDF at 23 wavelengths spread across the UV, VIS, NIR, SWIR, MWIR and LWIR regions. In addition, the DHR is extracted spectrally and fused with the BRDF in a similar manner as the built-in NEF calculations. As a result we have found that most materials can be captured in a data representation that only requires 2-10 MB per material.

Below is an outline of the workflow:

  • The Java-based nef2raw tool produces two files for each exported material:

    • The <name>.raw file is the sampled BRDF

    • The <name>.dhr file is the extracted DHR

  • The DIRSIG5 raw2sqt utility converts the <name>.raw file into the binary <name>.sqt file.

  • The <name>.sqt and <name>.hdr files are provided to DIRSIG5 in the material’s reflectance description.

Requirements

To use this workflow, the following software components are required to be installed:

  • The NEF database

    • This tutorial was created using NEF 16.1.

  • The Java Development Kit (JDK)

    • The "Standard Edition" JDK is available free of charge from the Oracle website.

    • This tutorial was created using Java 1.8.0 Standard Edition.

  • A DIRSIG5 installation

    • The Java-based nef2raw tool is included in the extras folder of all DIRSIG5 installations.

    • The raw2sqt tool is included in the bin folder of all DIRSIG5 installations.

Tutorial

The following tutorial will guide you through extracting a pair of materials from the NEF database and replacing the material properties in the Brdf2 demo.

Create a Working Folder

The "working folder" is the folder where the workflow will run. The location of this folder is not important as long as the user has write permissions for this folder. For the sake of this tutorial, we created a folder named nef2raw in the user’s desktop folder.

Copy and Update the Extraction Tool Files

Since you will need to modify the script for each material to extract, you will need to copy the nef2raw.java and nef2raw.bat file from the DIRSIG installation extras folder into your working folder. The nef2raw.java file should not need to be updated, but the nef2raw.bat file needs to be updated to (1) specify the path to your NEF installation and (2) change the material name (identifier) to be extracted. An example nef2raw.bat file is show below. The first line (@echo off) disables printing the command being executed to the output. The second line (starting with javac -cp) compiles the Java program for running. The third and final line (starting with java -cp) runs the program with the NEF "parameters" file as the first argument and the name of the material to be extracted as the second argument:

@echo off
javac -cp C:\NEFDS16110F_PC\NEFDS16_1.jar;. nef2raw.java
java -cp C:\NEFDS16110F_PC\NEFDS16_1.jar;. nef2raw C:\NEFDS16110F_PC\Properties\NEF_SQL_16_1.properties 1308UUUPNT
Note The Java program code does not change, so it is not necessary to recompile the program every time the program is to be used. However, this is a fairly quick process for this small program and it simplifies the example used here.
Note A more advanced .bat (or BASH script) could avoid hard coding the name of the material to be extracted.
Note On Linux, the Windows .bat file can be replaced by a BASH script that performs the same tasks.

You will need to edit the nef2raw.bat file to reflect the path to your NEF installation. Make sure the example path (e.g. C:\NEFDS16110F_PC ) is updated to reflect the path to the NEF installation for your computer. Furthermore, make sure the name of the "properties" file (in the Properties subfolder) reflects the version installed.

Update the Material Names to Extract

In the nef2raw.bat file, the third (final) command (starts with java -cp) is the one that runs the extraction program, and the 2nd (final) argument is the NEF name (identifier) of the material to be extracted. For this tutorial, we want to extract two materials: (1) a background material and (2) a target material. This could be accomplished by running the script as-is once for one material, editing the script to change the material and then running the script again for the other material. However, a simpler approach is to simply repeat the 2nd command for each material we want to extract.

For this tutorial, we will extract the following two materials:

0825UUUCNC

light gray construction concrete - USA(VA) - building

1455UUUPNT

glossy black paint on aluminum - USA - aluminum 3003 panel painted with oil-based glossy black paint (Rustoleum 7779-402)

Hence, the updated script looks like:

@echo off
javac -cp C:\NEFDS16110F_PC\NEFDS16_1.jar;. nef2raw.java
java -cp C:\NEFDS16110F_PC\NEFDS16_1.jar;. nef2raw C:\NEFDS16110F_PC\Properties\NEF_SQL_16_1.properties 0825UUUCNC
java -cp C:\NEFDS16110F_PC\NEFDS16_1.jar;. nef2raw C:\NEFDS16110F_PC\Properties\NEF_SQL_16_1.properties 1455UUUPNT

Extracting the Material Properties

With the script updated, the script can now be run to perform the extraction from the database:

C:\Users\dirsig\Desktop\nef2raw> .\nef2raw.bat
Property file: C:\NEFDS16110F_PC\Properties\NEF_SQL_16_1.properties
Connecting to database ...
Processing '0825UUUCNC'
Extracting the DHR:
    Min lambda = 0.25
    Max lambda = 13.9977
Extracting the BRDF:
    Extracting for source angle = 0.0
    Extracting for source angle = 10.0
    Extracting for source angle = 20.0
    Extracting for source angle = 30.0
    Extracting for source angle = 40.0
    Extracting for source angle = 50.0
    Extracting for source angle = 60.0
    Extracting for source angle = 70.0
    Extracting for source angle = 80.0
Disconnecting from database ...
Property file: C:\NEFDS16110F_PC\Properties\NEF_SQL_16_1.properties
Connecting to database ...
Processing '1455UUUPNT'
Extracting the DHR:
    Min lambda = 0.25
    Max lambda = 15.1613
Extracting the BRDF:
    Extracting for source angle = 0.0
    Extracting for source angle = 10.0
    Extracting for source angle = 20.0
    Extracting for source angle = 30.0
    Extracting for source angle = 40.0
    Extracting for source angle = 50.0
    Extracting for source angle = 60.0
    Extracting for source angle = 70.0
    Extracting for source angle = 80.0
Disconnecting from database ...
Note Addressing possible errors encountered by the script are beyond the scope of this tutorial. If errors are generated, carefully review the error messages and attempt to debug them.

After the extraction has run, your working folder should contain .raw and .dhr file pairs for both materials.

The working folder after extraction.

nef2raw folder

The .dhr files contain the extracted DHR for the two materials. These two curves are plotted below to show the differences in the magnitude between the bright concrete and dark black paint:

A plot of the DHR for the two extracted materials.

dhr plot

Convert the RAW files to SQT Files

The next step is to convert the ASCII/Text "raw" BRDF measurements into optimized, binary SQT files. This is performed by running the raw2sqt utility included with the DIRSIG software:

C:\Users\dirsig\Desktop\nef2raw> raw2sqt 0825UUUCNC.raw
Generating SQT file: "0825UUUCNC.sqt" ... done
C:\Users\dirsig\Desktop\nef2raw> raw2sqt 1455UUUPNT.raw
Generating SQT file: "1455UUUPNT.sqt" ... done

Updating the Brdf2 Demo Scene

The last step is to update the Brdf2 demo scene to use these two extracted materials. If you have not already downloaded that demo, then do so now. The remainder of this tutorial will assume you extracted the demo into the Brdf2 folder in your desktop folder.

  • ❏ Copy the .sqt and .dhr files into the materials folder inside the Brdf2 demo.

  • ❏ Edit the materials/demo.mat file

Graphically editing the material file to add the SQT reflectance property is a forthcoming feature. For now, the best approach is to edit the material file in a text editor to make the required changes. For this tutorial, we will be updating both materials in the scene. The first material (ID = 105) is the material applied to the "ground" and the second material (ID = test_mat) is applied to the vehicle. To make the changes, the following edits need to be made for each material:

  • ❏ The REFLECTANCE_PROP_NAME needs to be set to SphericalDataBRDF

  • ❏ In the REFLECTANCE_PROP section, the DATA_FILENAME needs to be assigned the name of the corresponding .sqt file

  • ❏ In the REFLECTANCE_PROP section, the DHR_FILENAME needs to be assigned the name of the corresponding .dhr file.

The example below shows the edits made to the 105 (ground) material:

MATERIAL_ENTRY {
    ID = 105
    NAME = 0825UUUCNC; light gray construction concrete
    EDITOR_COLOR = 1.0000, 1.0000, 1.0000
    DOUBLE_SIDED = TRUE

    SURFACE_PROPERTIES {
        REFLECTANCE_PROP_NAME = SphericalDataBRDF
        REFLECTANCE_PROP {
            DATA_FILENAME = 0825UUUCNC.sqt
            DHR_FILENAME  = 0825UUUCNC.dhr
        }
    }

    RAD_SOLVER_NAME = Simple
    RAD_SOLVER {
        QUALITY = LOW
    }
}

The test_mat material should be similarly updated, but assigned the corresponding .sqt and .dhr filenames for the 1455UUUPNT material:

MATERIAL_ENTRY {
    ID = test_mat
    NAME = 1455UUUPNT;glossy black paint on aluminum
    EDITOR_COLOR = 0.5, 0.5, 0.5
    DOUBLE_SIDED = TRUE

    SURFACE_PROPERTIES {
        REFLECTANCE_PROP_NAME = SphericalDataBRDF
        REFLECTANCE_PROP {
            DATA_FILENAME = 1455UUUPNT.sqt
            DHR_FILENAME  = 1455UUUPNT.dhr
        }
    }

    RAD_SOLVER_NAME = Simple
    RAD_SOLVER {
        QUALITY = LOW
    }
}
Note In an ideal world, the suggested material ID's would be NEF0825UUUCNC and NEF1455UUUPNT. However, that would require the material IDs (labels) to be updated in the Brdf2 geometry, which we felt was beyond the scope of this tutorial.

Running the Updated Brdf2 Demo

With the scene updates completed, the scene can be run. For DIRSIG5, recompile the scene with scene2hdf and then run the simulation:

C:\Users\dirsig\Desktop\Brdf2> scene2hdf demo.scene
C:\Users\dirsig\Desktop\Brdf2> dirsig5 demo.sim

The image below shows the output of the updated simulation. It should be noted that the simulation was updated to use a 640 x 480 camera (versus the original 320 x 240 camera) and the image below was created using a gamma = 2.0 scaling (e.g. envitopnm --autoscale=gamma --gamma=2 ...). Note the concrete color of the background and the glossy character of the black paint on the target vehicle:

The updated Brdf2 demo (gamma=2 scaling).

brdf2 nef

The image below is a similar rendering of the original Brdf2 material configuration:

The original Brdf2 demo (gamma=2 scaling).

brdf2 old