This document discusses the verification and validation (V&V) efforts related to DIRSIG.

Previous DIRSIG V&V Efforts

This section describes all the previous V&V projects that have been conducted on the DIRSIG model. Some of these efforts address end-to-end performance for a very specific application and others are very small in scope.

End-to-End, Broad-Band LWIR Validation (1992-1994)

Donna Rankin (M.S. student, now Donna Rankin-Parobek) and John Mason (staff, now at ITT SDD) did the "Rooftop Validation" experiment that was published on in 1994. In this effort we collected MWIR and LWIR imagery of a static scene (parking lot next to the CIS building) for 24 hours (images every hour). There was a weather station in the scene and thermistors recording the temperatures of 12 objects.

Analysis Performed:

  • Absolute temperature comparisons (THERM vs. thermistors)

  • Absolute apparent temperature comparisons in both the MWIR and LWIR

  • Relative or "Contrast Ranked" comparisons in both the MWIR and LWIR

Notable Results:

  • An important sensitivity analysis was performed to determine which model parameters are most important.

  • The impact of optimal vs. generic weather and material properties was found to be important.

  • Discovered that the scene geometry is very important, as was not measured well enough.

  • Identified limitations of thermal model knowledge of backgrounds and environment.

Related Publications:

  • Rankin, D., Salvaggio, C., Gallagher, T., Schott, J., "Instrumentation and procedures for validation of synthetic infrared image generation (SIG) models," Proc. SPIE, Infrared Technology XVIII, Vol. 1762, pp. 584-600, San Diego, CA, August 1992

  • Mason, J.E., Schott, J.R., and Rankin-Parobek, D., "Validation analysis of the thermal and radiometric integrity of RIT’s synthetic image generation model, DIRSIG", Proc. SPIE, Characterization and Propagation of Sources and Backgrounds, Vol. 2223, pp. 474-487. Orlando, FL 1994.

  • Mason, J.E., Schott, J.R., Salvaggio, C., Sirianni, J.D., "Validation of contrast and phenomenology in the Digital Imaging and Remote Sensing (DIRS) lab’s Image Generation (DIRSIG) model", Proc. SPIE, Vol. 2269, #72, San Diego, CA 1994.

  • Rankin. D., "Validation of the DIRSIG, An Infrared Synthetic Scene Generation Model", M.S. Thesis, Rochester Institute of Technology, February 1992

End-to-End, Broad-Band LWIR Validation, Revisited (1995)

The Rankin and Mason analysis was repeated in 1995/1996 by Scott Brown, Frank Tantalo and John Schott after the recommendations from the initial study were incorporated. Specifically, the scene geometry was updated to more precisely match the actual scene (primarily object orientations relative to the Sun). Improvements to the data supplied to the thermal model were made to address missing environmental influences.

Analysis Performed:

  • Absolute temperature comparisons (THERM vs. thermistors)

  • Absolute apparent temperature comparisons in both the MWIR and LWIR

  • Relative or "Contrast Ranked" comparisons in both the MWIR and LWIR

Notable Results:

  • The improvements to the thermal model allowed less optimal materials to perform better than before.

  • The scene geometry is still critically important, especially at transition times (sunrise and sunset).

Related Publications:

  • Brown, S.D., Schott, J.R., Raqueno, R.V., Kraska, T., White, R., "Validation and Analysis of the Digital Imaging and Remote Sensing Laboratory’s Synthetic Image Generation Model, DIRSIG", Proc of the Ground Target Modeling and Validation Conference, Houghton, MI, August 1996

Sensor/Platform Geometry Verification (1995)

Jim Salicain (M.S. student) and Scott Brown (staff) did a basic camera geometry verification as part of the implementation of the new focal plane and platform model.

Analysis Performed:

  • Simple photogrammetric comparisons of real and simulated data of the Rochester Urban/Hawkeye scene.

Notable Results:

  • Satisfactory agreement of geometric projections and distortions between real and synthetic imagery.

Related Publications:

  • Salacain, J., "DIRSIG Imaging System Geometric Models", M.S. Thesis, Rochester Institute of Technology, March 1995.

  • Salacain, J., Brown, S.D., Raqueno, R.V., Schott, J.R., "Incorporation of sensor geometry effects in synthetic image generation models", Proc. of the Ground Target Modeling and Validation Conference, Houghton, MI, pp. 420-434, August 1995

Solar and Emissive Spectral Phenomenology Validation (1996)

David Joseph (M.S. student) did a study comparing DIRSIG to data collected with the HYDICE Vis/NIR/SWIR hyper-spectral instrument and the SEBASS LWIR hyper-spectral instrument (first generation, LWIR only at that time). There was limited ground truth available for the scene and very little atmospheric characterization was performed. This study explored the questions about how to quantify the accuracy of predicted spectral signatures.

Analysis Performed:

  • Qualitative evaluation of absorption features.

  • Radiance rank order correlation (ROC) coefficients on a limited set of targets on a band-by-band basis.

Notable Results:

  • Qualitative spectral correlation was good.

  • The model had average ROC coefficients across all bands of 0.90+.

  • Atmospheric conditions was one of the biggest factors and was largely unknown.

  • SEBASS clearly had spectral calibration issues (known absorption features at the wrong wavelengths) that likely affected the results.

Related Publications:

  • Joseph, D., "DIRSIG: A Broadband Validation and Evaluation of Potential for Infrared Imaging Spectroscopy", M.S. Thesis, Rochester Institute of Technology, March 1998.

  • Brown, S.D., Joseph, D., Schott, J.R., and Raqueno, R., "Evaluation of DIRSIG for solar and emissive region hyperspectral image simulations," Proc of the Ground Target Modeling and Validation Conference, Houghton, MI, August 1998

Airborne, Broad-Band Visible and LWIR Validation (1996)

Russell White (M.S. student, U.S. Air Force) and Todd Kraska (M.S. student, U.S. Air Force) teamed up to validate DIRSIG against an improved and expanded version of the Urban/Hawkeye scene. The truth imagery was collected with an RGB digital camera, a broad-band LWIR framing camera collected from an lightweight airborne collection (summer) and some historical LWIR data collected by an LWIR line-scanner (winter).

Analysis Performed:

  • Rank order correlation (ROC)

Notable Results:

  • Limited to only a single, summer, daytime collection (visible) and the single, winter, night LWIR line-scanner (LWIR).

  • Decent ROC coefficients on a band-by-band basis. Need calibration for absolute radiometry.

Related Publications:

  • White, R., "Validation of Rochester Institute of Technology’s (RIT’s) Digital Imaging and Remote Sensing Image Generation (DIRSIG) Model - Reflective Region", M.S. Thesis, Rochester Institute of Technology, July 1996.

  • Kraska, T., M.S. Thesis, Rochester Institute of Technology, August 1996.

  • Brown, S.D., Schott, J.R., Raqueno, R.V., Kraska, T., White, R., "Validation and Analysis of the Digital Imaging and Remote Sensing Laboratory’s Synthetic Image Generation Model, DIRSIG", Proc of the Ground Target Modeling and Validation Conference, Houghton, MI, August 1996

Visible Low-Light Verification (1999)

Emmett Ientilucci (M.S. student) and Scott Brown added the support for moonlight, background night sky, point sources and extended area sources. Emmett’s thesis externally evaluated the implementation of the new low-light (low SNR) capabilities in DIRSIG as part of his larger scope.

Analysis Performed:

  • Comparison of DIRSIG implementation against analytical solutions for single sources.

  • Comparison of predicted night sky background fluxes to accepted literature.

  • Analysis of solar and lunar ephemeris model.

Notable Results:

  • Verification of point sources radiometry against analytical solutions.

  • Verification of extended area sources radiometry against analytical solutions.

  • Verification of night sky background flux.

  • Analysis of solar and lunar ephemeris.

Related Publications:

  • Ientilucci, E. "Synthetic Simulation and Modeling of Image Intensified CCDs (IICCDs)", M.S. Thesis, Rochester Institute of Technology, July 1999

  • Ientilucci, E.J., Brown, S.D., Schott, J.R., Raqueno, R.V., "Multi-spectral simulation environment for modeling low-light-level sensor systems", Proc. SPIE, San Diego, July 1998

V/NIR/SWIR Spectral Texture Validation (2003)

Niel Scanlan (M.S. student, Canadian Air Force) did a comparative analysis of the different texture methodologies available in the DIRSIG model. These methodologies attempt to add the spatial and spectral variability or "texture" observed in most "backgrounds" in the scene. Of specific interest was the ability of these methods to reproduce both the spatial and the spectral statistics of the scene.

The method available in the DIRSIG model include the following:

  • The traditional texture technique using a single band to drive curve selection.

  • The improved texture technique using a multiple bands to drive curve selection.

  • The material mixing approach that remixes a set of materials based on the result of an ummixing process.

  • The in-house, spatial-spectral synthesis technique.

Analysis Performed:

  • Spatial statistics including the Grey Level Co-Occurance Matrix (GLCM) as a function of spectral band.

  • Spectral statistics including the Signal-to-Clutter Ratio (SCR) metric and Spectral Co-Occurrence Matrix (SCM) metric.

Notable Results:

  • The traditional DIRSIG texture methodology provides the best flexibility and performance. Using more uncorrelated bands improves performance.

  • Mixture maps had the best spatial and spectral performance, but it is nearly an empirical technique. Hence, you need to have data with the nearly the same spatial and spectral fidelity to use this method. If you had it, why would you need to simulate it.

  • The in-house, spatial-spectral synthesis technique was very good spectrally but poor spatially. It needs large spatial and spectral training sets to work well.

Related Publications:

  • Scanlan, N.W., Schott, J.R., Brown, S.D., "Performance analysis of improved methodology for incorporation of spatial/spectral variability in synthetic hyperspectral imagery", Proc. SPIE, Vol. 5159, p. 319-330, August 2003

  • Scanlan, N.W., "Comparative Performance Analysis of Texture Characterization Models in DIRSIG", M.S. Thesis, Rochester Institute of Technology, September 2003

Airborne LWIR Spectral Simulation Validation (2004)

Erin Peterson (M.S. student, U.S. Air Force) worked on a project to created a simulated desert scene containing buried landmines. This scene was intended to be a simulated analog of a real site that was created that Yuma Proving Grounds (YPG) and collected with the University of Hawaii’s Airborne Hyperspectral Imager (AHI). The key challenge of this project was that the real data was collected months before the synthetic scene was constructed and very little ground truth was collected at the actual experiment. The project was dominated by learning how to incorporate spatial spectral variability (texture) without access to ground truth data.

Analysis Performed:

  • Calibrated image histograms to evaluate overall variability as a function of time of day (per band).

  • Rank order correlation (ROC) of materials in the scenes (per band).

  • Qualitative radiance spectrum comparisons (magnitude, shape, etc.)

  • Spectral dimensionality analysis.

  • Spatial and spectral statistics (GLCM and SCM metrics).

  • Performance of Rx on real vs. synthetic datasets.

Notable Results:

  • Good overall spatial, spectral temporal variability agreement.

  • This is a case where algorithm performance (e.g. Rx) was a primary metric.

  • Difficult to simulate scenes where no ground data is available.

Related Publications:

  • Peterson, E.D., Brown, S.D., Hattenberger, T.J. and Schott, J.R., "Surface and buried landmine scene generation and validation using the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model", Proc. SPIE, Vol. 5546, p. 312-323, Denver CO, August 2004

  • Peterson, E.D., "Synthetic Landmine Scene Development and Validation in DIRSIG", M.S. Thesis, Rochester Institute of Technology, 2004.

High-Resolution, Slant-Look Concealed Targets Validation (2004)

Kris Barcomb (M.S. student, U.S. Air Force) constructed a new scene (now known as MicroScene #1) to explore the ability of DIRSIG to create high-resolution, slant-look, visible imagery for testing detection of concealed targets (landmines, camouflage, etc.). The scene was on the RIT campus which (unlike the Peterson study conducted at roughly the same time) provided the investigators which unlimited access to the base site (although the deployed objects were no longer present). This scene was limited in spatial coverage, but was constructed at a very, very high spatial resolution (individual blades of grass).

The corresponding data collection experiment was a 24-hour effort that included in-scene thermistors and imagery collected by the RIT WASP sensor (a suite of Vis, SWIR, MWIR and LWIR framing cameras) and the RIT MISI sensor (a Vis/NIR imaging spectrometer with broad band LWIR channels).

Analysis Performed:

  • Qualitative image analysis.

  • Grey-Level Co-Occurance Matrix (GLCM) texture metrics.

  • Performance of Gaussian Maximum Likelihood (GML) on real and synthetic imagery.

  • Performance of Rx on real and synthetic imagery.

Notable Results:

  • This study included a spiral development approach to identify limitations of the model and address them.

  • This study utilized algorithm performance (GML and Rx) as a primary metric.

  • Good performance agreement of GML and Rx. Potential issues with sky character at transition times.

Related Publications:

  • Barcomb, K.E., Brown, S.D., Hattenberger, T.J. and Schott, J.R., "High-resolution slant-angle scene generation and validation of concealed targets in DIRSIG", Proc. SPIE, Vol. 5546, p. 300-311, Denver CO, August 2004

  • Barcomb, K., "High-Resolution, Slant-Angle Scene Generation and Validation of Concealed Targets in DIRSIG", M.S. Thesis, Rochester Institute of Technology, 2004.

Visible Polarization Verification (2005)

James Shell II (Ph.D. student, U.S. Air Force) worked on creating a polarized BRDF models that could be used for (a) targets and (b) backgrounds (including variability). His work resulted in two pBRDF models that are currently called "Shell Target" and "Shell Background". The "Shell Target" model is a generalized micro-facet distribution model that can emulate many of the common models with this heritage (Cook-Torrance, Torrance-Sparrow, Beard-Maxwell, etc.). The "Shell Background" model is unique in that it accounts for both directional and spatial (texture) variability.

Although his work was largely on the implementation and demonstration, he performed some verification studies.

Analysis Performed:

  • First and second order image statistics of polarized images (Stokes "bands") and polarized data products (degree of polarization, angle of polarization, etc.) from a given view angle.

Notable Results:

  • Image based characterization technique could reproduce first order angular (pBRDF) and spatial (texture) characteristics of the original background material.

Related Publications:

  • Shell, J. and Schott, J.R., "A polarized clutter measurement technique based on the governing equation for polarimetric remote sensing in the visible to near infrared", Proc. SPIE, Vol. 5811, pp.34-45, May 2005.

  • Shell, J., "Polarimetric Remote Sensing in the Visible to Near Infrared", Ph.D. dissertation, Rochester Institute of Technology, 2005.

LIDAR/LADAR Verification (2005)

Daniel Blevins (Ph.D. student, U.S. Air Force) and Scott Brown did a series of verification studies related to active laser sensing. These studies were a portion of Dan’s larger body of work looking at the impact of scattering on Differential Absorption LIDAR (DIAL).

Analysis Performed:

  • Temporal pulse skew from a tilted surface.

  • Analytical verification of a return from a flat plate.

  • Experimental verification experimental validation of a returns from a flat plate.

  • Time and flux statistics of multiple scattering in clouds, plumes, etc.

Notable Results:

  • Verified correct behavior of temporal pulse skew from a tilted surface.

  • Analytical verification of a return from a flat plate.

  • Experimental validation of a return from a flat plate using data from the MIT/LL ALIRT sensor.

  • Reproduced temporal flux statistics of multiple scattering against other models.

Related Publications:

  • Brown, S.D., Blevins, D.D., and Schott, J.R., "Time-gated topographic LIDAR scene simulation", Proc. SPIE, Vol.5791, p. 342-353, Orlando FL, April 2005

  • Blevins, D.D., "Modeling Scattering and Absorption for a Differential Absorption LIDAR System", Ph.D. Dissertation, Rochester Institute of Technology, 2005

  • Blevins, D.D., Brown, S.D., and Schott, J.R., "First-principles based LIDAR simulation environment for scenes with participating mediums", Proc. SPIE, Vol. 6214, 62140G, Orlando FL, May 2006

Visible Polarization Validation (2006)

In 2007, two small efforts were performed to verify and validate the basic principles of the polarization modality. These were small and largely internally funded efforts, so the scope was very limited. The key areas of interest were (a) integration with MODTRAN-P, (b) global-to-surface and surface-to-global Stokes orientation rotations and (c) the fundamentals of basic outdoor scene simulation.

Analysis Performed:

  • Comparison of DIRSIG using MODTRAN-P with Coulson field data.

  • Qualitative evaluation of surface orientation and illumination effects in the lab.

  • Exploration of polarization clutter in a real work scene.

Notable Results:

  • DIRSIG using MODTRAN-P agreed with the degree of polarization (DoP) and angle of polarization (AoP) Coulson field data.

  • The trends in the degree of polarization (DoP) and angle of polarization (AoP) in the lab experiment were consistent with well understood trends (DoP increased for off axis angles, AoP magnitude and sign was correct).

  • The use of high geometric resolution features is required to reproduce some polarization clutter. In this case, the grooves in the milled parking lot were a significant source of clutter.

Related Publications:

  • Devaraj, C., Brown S., Messinger, D., Goodenough, A., Pogorzala D., "A framework for polarized radiance signature prediction for natural scenes", Proc. SPIE, Vol. 6565, 65650Y, May 2007

  • Pogorzala D., Brown S., Messinger, D., Goodenough, A., Devaraj, C., "Recreation of a nominal polarimetric scene using synthetic modeling tools", Proc. SPIE, Vol. 6565, 65650Z, May 2007

Thermal Polarization Verification (2007)

Michael Gartley (Ph.D. student, now staff) worked on DIRSIG’s ability to do thermal polarization. As part of that effort, Mike did a few "low budget" experiments to aid in basic qualitative phenomenology checks. He collected data outside at night using an LWIR camera and a wire grid polarizer.

Analysis Performed:

  • Qualitative phenomenology effects between targets and backgrounds at various temperatures.

  • Degree of polarization (DoP) and angle of polarization (AoP)

Notable Results:

  • Demonstrated correct angle of polarization (AoP) for surface emitted radiance as a function of surface to camera relative angles using a painted beach ball.

  • Demonstrated correct phenomenology for targets and backgrounds at various temperatures using an scene with an array of targets and a warm background that could be moved in and out of the scene.

Related Publications:

  • Gartley, M., "Polarimetric Modeling of Remotely Sensed Scenes in the Thermal Infrared", Ph.D. Dissertation, Rochester Institute of Technology, May 2007

  • Gartley, M., Brown S., Goodenough, A., Sanders, N., Schott, J., "Polarimetric scene modeling in the thermal infrared", Proc. SPIE, Vol. 6682, 66820C, September 2007