Overview

The output of a DIRSIG Lidar simulation is a binary file that contains a series of spatial x spatial x time cubes for each pulse with some platform ephemeris data mixed in. Because this file contains the number of photons arriving at each detector during the set of time bins determined by the range gate defined for the simulation, these files are sometimes called "bin" files.

This output file is usually feed to a detector model which "senses" when the detector will mark a "return" based on the incident flux for a given pixel. This detection model might find multiple returns within one pulse (common in most Linear-mode systems) or only a single-return (common in most Geiger-mode systems). Regardless of the detector model applied, the output of DIRSIG is the same.

Note This document describes the raw photon arrival data computed by DIRSIG. It does not describe a point cloud data format that might be generated by a detector model that takes this file as an input.

Tracking Changes

This document emphasizes the current format of the "bin" file. The descriptions of each section include all entries since the first version of this file was defined. Deprecated entries (obsolete entries that have been removed) are struck through, although they remain in the document for reference. The History section near the end of this document summarizes the changes for each revision.

Format Details

File Structure

The general format of the BIN file is as follows:

  • File Header

    • Task Header

      • Pulse Header

      • Pulse Data

There is always only one File Header in a file. There may be multiple tasks in a file, each starting with a Task Header. Within a task, there may be multiple pulses, each starting with a Pulse Header that is immediately followed by the Pulse Data. Note that the File Header indicates the number of tasks contained in the file and the Task Header indicates the number of pulses contained in that task.

See below for a more complete example of how a larger file is arranged:

  • File Header (indicates there are N tasks in this file)

    • Task Header #1 (indicates there are M1 pulses in this task)

      • Pulse Header #1

      • Pulse Data #1

      • Pulse Header #2

      • Pulse Data #2

      • Pulse Header #3

      • Pulse Data #3

      • …​

      • Pulse Header #M1

      • Pulse Data #M1

    • Task Header #2 (indicates there are M2 pulses in this task)

      • Pulse Header #1

      • Pulse Data #1

      • Pulse Header #2

      • Pulse Data #2

      • …​

      • Pulse Header #M2

      • Pulse Data #M2

    • …​

    • Task Header #N (indicates there are M3 pulses in this task)

      • Pulse Header #1

      • Pulse Data #1

      • Pulse Header #2

      • Pulse Data #2

      • …​

      • Pulse Header #M3

      • Pulse Data #M3

Multiple Files

A single DIRSIG simulation might produce multiple output BIN files. That is because the user has three (3) options available related to the output file schedule:

  • A single BIN file for the whole simulation where all pulses for all tasks appear in one file. For example lidar.bin.

    • In this situation, the BIN file will contain multiple tasks, and each task can contain multiple pulses.

  • Individual BIN files for each task, where all the pulses for a given task appear in one file. The output files will have a unique file name for each task. For example lidar-t0000.bin, lidar-t0001.bin, etc.

    • In this situation, the each BIN file will contain a single task, but each task can contain multiple pulses.

    • The "file header" will be the same in every file.

  • Individual BIN files for each pulse, where each pulse appears in it’s own file. The resulting files with have a unique file name for each pulse and each task. For example, lidar-t0000-c0000.bin, lidar-t0000-c0001.bin, …​ lidar-t0001-c0000.bin, lidar-t0001-c0001.bin, …​, etc.

    • In this situation, each BIN file will contain a single task containing a single pulse.

    • The "file header" will be the same in every file.

    • The "task header" will be the same for all pulse files from the same task.

The detector model that processes these file can choose to merge the returns for all these files into one output product.

Data Types

The following section outlines the data types used in the various sections of this file:

Type Description

int8

8 bits, signed via two’s complement, -128 to +127

int32

4 bytes, signed via two’s complement, -2,147,483,648 to +2,147,483,647

uint16

2 bytes, unsigned, 0 to +65,535

uint32

4 bytes, unsigned, 0 to +4,294,967,295

uint64

8 bytes, unsigned, 0 to +18,446,744,073,709,55

double

8 bytes, IEEE754 format
Note Data is written in native endian of the host machine. The File Header contains the Byte ordering variable that indicates if the host machine was big endian (Most Significant Byte First (MSF)) or little endian (Least Significant Byte First (LSF)).

File Header

This section describes the data layout of the File Header section of the BIN file. There is exactly one File Header section and it appears at the start of the file.

File Identifier

char[11]

File format revision

int8

Byte ordering

int8

File creation date/time

char[15]

DIRSIG version string

char[32]

Simulation description

char[256]

Scene origin latitude

double

Scene origin longitude

double

Scene origin height

double

Transmitter mount type

char[16]

Receiver mount type

char[16]

Pixel count, X dimension

uint32

Pixel count, Y dimension

uint32

Pixel pitch, X dimension

double

Pixel pitch, Y dimension

double

Array offset, X dimension

double

Added in [v.1]

Array offset, Y dimension

double

Added in [v.1]

Lens distortion coefficient (k1)

double

Added in [v.1]

Lens distortion coefficient (k2)

double

Added in [v.1]

Task count

uint32

Focal plane array (FPA) ID

uint16

Added in [v.2]

The total size of this header is currently 434 bytes ([v.1]: 432 bytes [v.0]: 400 bytes).

File Identifier

Initial bytes always read "DIRSIGPROTO", used to verify file type.

File format revision

Version of file format specification. Currently 0 (DIRSIG 4.3.0), 1 (DIRSIG 4.3.1 - 4.4.4), and 2 (DIRSIG 4.5 and later)

Byte ordering

0 indicates big endian, 1 for little endian.

File creation date/time

The date and times for when the file was created (Format: YYYYMMDDhhmm.ss).

DIRSIG version

The DIRSIG version used to create file. Unused characters at end should be filled with nulls (ASCII code 0)

Simulation description

An arbitrary user-supplied text describing simulation, primarily used for documentation purposes. Unused characters at end should be filled with nulls (ASCII code 0)

Scene origin latitude

Latitude of scene origin in degrees [-90,90], where negative is South.

Scene origin longitude

Longitude of scene origin in degrees [-180,180), where negative is West.

Scene origin height

Height of scene origin in meters above WGS84 reference ellipsoid.

Transmitter mount type

A string description of mount type for laser source (transmitter), primarily used for documentation purposes. Unused characters at end should be filled with nulls (ASCII code 0).

Receiver mount type

A string description of mount type for detector (receiver), primarily used for documentation purposes. Unused characters at end should be filled with nulls (ASCII code 0).

Pixel count, X dimension

Number of detector elements in the X dimension of the array.

Pixel count, Y dimension

Number of detector element in the Y dimension of the array.

Pixel pitch, X dimension

Distance between detector centers in microns, for the X dimension of the array.

Pixel pitch, Y dimension

Distance between detector centers in microns, for the Y dimension of the array.

Array offset, X dimension

Distance between array center and optical axis in microns.

Array offset, Y dimension

Distance between array center and optical axis in microns.

Lens distortion coefficient, k1

Radial lens distortion coefficient.

Lens distortion coefficient, k2

Radial lens distortion coefficient.

Task count

The number of tasks contained in the file.

Focal Plane Array (FPA) ID

Unique ID for the focal plane array used to create this file. FPAs are numbered in the order they appear in the instrument, starting at 0. The FPA IDs are not unique across multiple instruments.

Task Header

This section describes the data layout of the Task Header section of the BIN file. There is one Task Header section at the start of each task block within the file.

Task description

char[64]

Task start date/time

char[15]

Task stop date/time

char[15]

Focal length

double

Pulse repetition frequency

double

Pulse duration

double

Pulse energy

double

Laser spectral center

double

Laser spectral width

double

Pulse count

uint32

The total size of this header is currently 146 bytes.

Task description

An arbitrary text string describing task, primarily used for documentation purposes. Unused characters at end should be filled with nulls (ASCII code 0).

Task start date/time

The date and timestamp for the start of this task (Format: YYYYMMDDhhmm.ss).

Task stop date/time

The date and timestamp for the stop (end) of this task (Format: YYYYMMDDhhmm.ss).

Focal length

The focal length of the receiver in millimeters.

Pulse repetition frequency

The mean pulse repetition rate in Hertz.

Pulse duration

The temporal width of the transmitted laser pulse in seconds (assumed Gaussian).

Pulse energy

The energy of each transmitted laser pulse in in Joules.

Laser spectral center

The spectral center of the transmitted laser pulse in microns.

Laser spectral width

The spectral width of the transmitted laser pulse in microns (assumed Gaussian).

Pulse count

The number of pulses in this task.

Pulse Header

This section describes the data layout of the Pulse Header section of the BIN file. There is one Pulse Header section at the start of each pulse block within the file.

Pulse time

double

Time gate start

double

Time gate stop

double

Time gate bin count

uint32

Samples per time bin

uint32

Added in [v.1]

Platform location

double[3]

Platform orientation: Angle order

char[3]

removed in [v.2]

Platform rotation

double[3]

Transmitter to mount affine

double[16]

Added in [v.2]

Transmitter mount pointing offset

double[3]

removed in [v.2]

Transmitter orientation: Angle order

char[3]

removed in [v.2]

Transmitter mount pointing rotation

double[3]

Transmitter mount to platform affine

double[16]

Added in [v.2]

Receiver to mount affine

double[16]

Added in [v.2]

Receiver mount pointing offset

double[3]

removed in [v.2]

Receiver orientation: Angle order

char[3]

removed in [v.2]

Receiver mount pointing rotation

double[3]

Receiver mount to platform affine

double[16]

Added in [v.2]

Pulse data type

int32

Data compression type

int8

Delta histogram flag

char

removed in [v.2]

Pulse index

uint32

Added in [v.2]

Pulse data bytes

uint64

Guaranteed to be 64-bits in [v.2]

System transmit Mueller matrix

double[16]

Added in [v.2]

System receive Mueller matrix

double[16]

Added in [v.2]

The total size of this header is currently 913 bytes ([v.1]: 199 bytes, [v.0]: 192 bytes).

Pulse time

Pulse time relative to task start [seconds].

Time gate start

Time gating start time relative to pulse start [seconds].

Time gate stop

Time gating stop time relative to pulse stop [seconds].

Time gate bin count

Number of time bins (equally spaced, temporal samples) over range gate.

Samples per time bin

The number of samples per time bin. This value can also be interpreted as a multiplier for Time gate bin count.

Platform location

The platform location in the Scene ENU coordinate frame [meters].

Platform rotation

The Euler angles for the orientation of the platform in the Scene ENU coordinate frame [radians]. The data is stored in "XYZ" order, but the angles must be applied in "YZX" order.

Transmitter to mount affine

Affine transform (4x4, row-major order) placing the transmitter relative to its respective mount (this represents the instrument to mount attachment).

Transmitter mount pointing rotation

The Euler angles for the rotation of the transmitter mount [radians]. The data is stored in "XYZ" order, but the angles must be applied in "YZX" order.

Transmitter mount to platform affine

Affine transform (4x4, row-major order) placing the transmitter mount relative to the platform (this represents the transmitter mount to platform attachment).

Receiver to mount affine

Affine transform (4x4, row-major order) placing the receiver relative to its respective mount (this represents the instrument to mount attachment).

Receiver mount pointing rotation

The Euler angles for the rotation of the receiver mount [radians]. The data is stored in "XYZ" order, but the angles must be applied in "YZX" order.

Receiver mount to platform affine

Affine transform (4x4, row-major order) placing the receiver mount relative to the platform (this represents the receiver mount to platform attachment).

Pulse data type

The data type used in pulse results with only type 5 (double precision floating-point) being currently supported. This value corresponds to the "data type" values defined in the ENVI image header file.

Data compression type

0 indicates no compression, 1 indicates zlib compression of the pulse data.

Pulse index

The index of this pulse in the current task (indexes start at 0).

Pulse data bytes

The number of bytes of (possibly compressed) pulse data. If the data is not compressed, the size should be X x Y x (T + 1).

System transmit Mueller matrix

A Mueller matrix that captures all the polarizing elements between the transmitter exit aperture (assumed to be unpolarized) and the exit point of the platform, including scan mirrors, glass, polarization state generators, etc.

System receive Mueller Matrix

A Mueller matrix that captures all the polarizing elements between the entrance point of the platform and the focal plane array (FPA), including scan mirrors, glass, filters, polarization state analyzers, etc.

Pulse Data

The pulse data contains the number of photons received in each listening window (range gate) time bin, for each pixel in the receiver. The pulse data separately contains the background (passive) flux as a rate (units of photons/second). This passive flux appears in the first "time bin" of the pulse data. Therefore, to get the total number of photons arriving within a given time bin, the background (passive) value in the first "bin" (the passive "bin") should be multiplied by the width of a time bin and added to the value in each remaining bin (the "active" bins).

Note The total number of "active" bins is the Time gate bin count multiplied by the Samples per time bin.

From a data dimensionality standpoint, the pulse data always contains the number of "active" time bins plus an extra bin for the background (passive) flux.

Note The "extra" bin containing the background (passive) flux is not included in the value stored in the Time gate bin count field.

The 3D data cube is packed into the 1D data stream of the file in the order time, then X, then Y ("time interleaved by pixel"). For example:

(X0,Y0,Bkg)     (X0,Y0,T0)     (X0,Y0,T1)     ... (X0,Y0,Tmax)
(X1,Y0,Bkg)     (X1,Y0,T0)     (X1,Y0,T1)     ... (X1,Y0,Tmax)
...
(Xmax,Y0,Bkg)   (Xmax,Y0,T0)   (Xmax,Y0,T1)   ... (Xmax,Y0,Tmax)
(X0,Y1,Bkg)     (X0,Y1,T0)     (X0,Y1,T1)     ... (X0,Y1,Tmax)
(X1,Y1,Bkg)     (X1,Y1,T0)     (X1,Y1,T1)     ... (X1,Y0,Tmax)
...
(Xmax,Ymax,Bkg) (Xmax,Ymax,T0) (Xmax,Ymax,T1) ... (Xmax,Ymax,Tmax)
Note The background (temporally constant) bin has units of photons/second and the remaining (temporally varying) bins have units of photons.

File Parsing Guidance and Tools

The following is a set of hints and suggestions for reading BIN files:

  • Each BIN file can have a unique number of tasks, and each task within a file can have a unique number of pulses.

  • The pulse data includes all bins in the range gate inclusively.

Example MATLAB Code

Each DIRSIG release includes MATLAB code for reading the BIN file. This code can be used as a template to write your own detector model or to perform analysis on a file.

This code is contained in the extras/matlab-lidar folder in the DIRSIG installation folder.

Computing Pixel Line of Sight Vectors

Computing pixel line-of-sight vectors that include all of the platform relative offset and pointing effects must consider the following information:

  • The transmitter and receiver are located and oriented relative to the platform coordinate system.

  • The transmitter and receiver are oriented based on the supplied angles. This rotation is about the translated location of the respective instrument. Therefore the order is rotate the instrument relative to the platform, then translate the instrument (again, relative to the platform).

The final line-of-sight vector is the result of applying a series of affine transforms to a camera-relative pixel vector. The various transforms change on a pulse-to-pulse basis due to platform motion and platform-relative pointing (scanning). The following instructions attempt to detail the chain of transforms employed in the computation:

  1. A given pixel’s camera-relative pointing vector can be computed using the pixel pitch and array offset in the File Header section and the focal length in the Pulse Header section.

    • Note that by default, DIRSIG cameras point down from the platform. The easiest way to achieve this is to use the negative of the focal length as the Z component of the pixel vector.

  2. The camera-relative vectors must now be modified by the mount pointing (scanning) transform. This is driven by the receiver mount pointing rotation angles in the corresponding Pulse Header section. The combined rotation transform can be computed by chaining the individual axis rotation transforms in the order Y, then Z, then X.

  3. The resulting mount-relative vectors must now be modified by the receiver mount to platform affine transform in corresponding Pulse Header section. This transform handles any spatial and angular offsets of the mount relative to the platform body.

  4. The platform-relative vectors must now be transformed into the Scene ENU coordinate system by accounting for the platform orientation and translation.

    1. The platform orientation transform is then applied using the platform rotation values in the Pulse Header section. The combined rotation transform can be computed by chaining the individual axis rotation transforms in the order Y, then Z, then X.

    2. Finally, the platform translation transform is applied using the platform location values in the Pulse Header. The combined translation transform can be computed by combining the individual axis translation transforms.

Constructing a 3D point from a time-of-flight

Level-1 (L1) LIDAR data is usually a 3D point cloud, where points represent a location in 3D space that is corresponding to a singal return for which a time-of-flight was measured. If you are implementing your own detector model (which produces a time-of-flight measurement) and want to construct a 3D point for that time-of-flight, the following process is suggested:

  1. Compute the pointing vector for the given pixel.

  2. Compute the absolute location of the camera using the sames steps to produce the pointing vector, but using the 3D origin point (0,0,0) rather than the camera-relative pixel pointing vector.

  3. Compute the time-of-flight for a given pixel using the waveform contained in the Pulse data.

  4. Compute the round-trip distance corresponding to that time-of-flight.

  5. Compute the platform-relative 3D point by projecting half the round-trip distance along the pixel pointing vector.

  6. Compute the absolute location of the 3D point by adding the absolute camera origin (described in step #2) to the platform-relative 3D point computed in the previous step.

Note Depending on the atmospheric configuration supplied to DIRSIG, the index of refraction along the path might vary, which results in the speed of light varying along the path.

History

This section details the history of this file format, including changes made between revisions (see the File format revision field in the File Header).

Revision 0

The initial version of this file.

Revision 1 Changes

Below is the list of changes that were introduced in Revision 1:

  • File Header

    • Added detector array X/Y offsets

    • Added radial lens distortion model coefficients (K1 and K2)

  • Pulse Header

    • Added "Samples per time bin" to pulse header

Revision 2 Changes

Below is the list of changes that were introduced in Revision 2:

  • File Header

    • Added the Focal plane array (FPA) ID

  • Pulse Header

    • Added affine matrices for the "instrument to mount" and "mount to platform" attachment transformations

    • Removed transmitter and receiver mount offsets (more accurately captured by the two new affine transforms)

    • Removed various Euler angle rotation order fields because the angles always follow "YZX" order

    • Removed deprecated (and unused) delta histogram flag

    • Added the Pulse index

    • Guaranteed that the Pulse data bytes will always be 64-bits (prior to v.2, it was 32-bits on a 32-bit system and 64-bits on a 64-bit system)

    • Added Mueller matrices for transmit and receive optical paths

Proposed Changes

Below is a proposed list of changes to made to the file at a future date:

  • File Header

    • Switch to ISO8601 for date/time strings (seconds precision issue)

    • Include support for other lens distortion models (some sort of ID and the number of bytes used to store the parameters for the model).

  • Task Header

    • Switch to ISO8601 for date/time strings (seconds precision issue)

    • Include the task index (this was supposed to go into rev. 2)

    • Include the "shape" of the temporal pulse profile and the spectral pulse shape. Right now these widths are assumed to correspond to a Gaussian shape, but the user might not have setup DIRSIG to use a Gaussian shape. Even if we included the name of the functional form used in the simulation the user would need to refer to the DIRSIG Beam Manual to know what the widths mean, and it would not address a user-defined profile. Including a 1D rendering of each profile would address all of these limitations.