There are currently a large number of scripts and libraries for DEM related tasks, most commonly used to:

• Create DEMs for use in APL
• Create DEMs from LiDAR data
• Perform general tasks on DEMs (e.g., reprojection)

These scripts are a complicated mix of bash and Python which are very difficult to debug. Previous attempts to simplify them have only increased the complexity.

There are problems with the existing scripts which are proving difficult to fix given the current state of the code, the biggest one is that the DEMs produced are often much larger than needed.

A rewrite/restructure of the existing DEM scripts is therefore proposed with the following aims:

• Use Python instead of bash where possible
• Store common functions within a Python library
• Make use of existing libraries both external (e.g., GDAL, Proj, GRASS Python bindings) and internal where possible.

Given these aims the following is proposed:

New DEM Scripts

create_apl_dem.py Create DEM subset to navigation data for use in APL. Locations of existing DEMs (ASTER and NMB) will be hardcoded but will allow any DEM mosaic (real or virtual raster) to be provided.

  -o Out DEM, --outdem Out DEM
                        Output name for DEM
  -n Nav file, --nav Nav file
                        Navigation data (.sol / .sbet file)
  -p Main project directory, --project Main project directory
                        Main project directory (default=".")
  --aster               Use ASTER data
                        (/users/rsg/arsf/aster/aster_15m_dem_mosaic.vrt)
  --nextmap             Use Nextmap data (/users/rsg/arsf/nextmap/neodc/nextma
                        p_dsm_mosaic_bng.vrt)
  --srtm                Use SRTM data (/local1/data/basedata/srtm/srtm_global_
                        mosaic_90m.tif)
  --demmosaic Input DEM mosaic
                        Input DEM mosaic. For non-standard DEM. Use "--aster"
                        or "--nextmap" for standard DEMs.
  --separation_file Seperation file
                        File with Height offset to add if "--demmosaic" is
                        used and DEM heights are not relative to WGS-84
                        elepsoid. Not required if using "--aster", "--nextmap"
                        or "--srtm" for standard DEMs.
  -b BIL Navigation Files, --bil_navigation BIL Navigation Files
                        Directory containing post-processed navigation files
                        in BIL format. By default raw navigation data will be
                        used for "--project". If this is not available (e.g.,
                        for ARSF delivered data use this option and point to
                        "flightlines/navigation" within delivery directory
  --keepgrassdb         Keep GRASS database (default=False)

create_dem_from_lidar.py Create DEM from lidar data

  -o Out DEM, --outdem Out DEM
                        Output name for DEM
  -s Out Screenshot File or Directory, --screenshot Out Screenshot File or Directory
                        Output directory for screenshots or single file for
                        screenshot of mosaic, in JPEG format.
  --las                 Input LiDAR data are in LAS format (default=True)
  --ascii               Input LiDAR data are in ASCII format (default=False)
  -r Resolution, --resolution Resolution
                        Resolution for output DEM (default=2)
  --in_projection In Projection
                        Input projection (e.g., UTM30N; default=UKBNG)
  --out_projection Out Projection
                        Out projection. Default is same as input
  -n Nav file, --nav Nav file
                        Navigation data (.sbet / .sol file) used if patching
                        with another DEM
  -p Main project directory, --project Main project directory
                        Main project directory, used if patching with another
                        DEM
  --demmosaic Input DEM mosaic
                        Input DEM mosaic to patch with in GDAL compatible
                        format. Vertical datum needs to be the same as output
                        projection. Only required for non-standard DEM. Use "
                        --aster" or "--nextmap" for standard DEMs.
  --aster               Patch with ASTER data
                        (/users/rsg/arsf/aster/aster_15m_dem_mosaic.vrt)
  --nextmap             Patch with Nextmap data (/users/rsg/arsf/nextmap/neodc
                        /nextmap_dsm_mosaic_bng.vrt)
  --hyperspectral_bounds
                        If patching with another DEM, get extent from
                        hyperspectral navigation data, recommended if DEM is
                        to be used with APL and navigation data are available.
  --lidar_bounds        If patching with another DEM, get extent from lidar
                        data plus default buffer of 2000 m. If DEM is not
                        required to be used with APL this option is
                        recommended.

New DEM Library Putting all the DEM functions in a separate module (arsf_dem) is proposed rather than multiple libraries.

|-- arsf_dem
|   |-- arsf_dem.cfg
|   |-- common_functions.py
|   |-- dem_common.py
|   |-- dem_lidar
|   |   |-- ascii_lidar.py
|   |   |-- grass_lidar.py
|   |   |-- __init__.py
|   |   |-- laspy_lidar.py
|   |   |-- lastools_lidar.py
|   |   |-- lidar_utilities.py
|   |   |-- spdlib_lidar.py
|   |-- dem_nav_utilities.py
|   |-- dem_utilities.py
|   |-- grass_library.py

Data location: ~arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_289_Danum_Valley

Data arrived from ARSF via network transfer on 16/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full waveform LiDAR data are present Thermal data are present

Some corrupt webcam images were deleted: 141016_004756_000.jpg 141016_004327_000.jpg 141016_013510_000.jpg 141016_005255_000.jpg 141016_005117_000.jpg 141016_015132_000.jpg

Sensors:

• Fenix (requested)
• Leica LIDAR (requested, delivered 27/04/2015)
• Leica LIDAR FW (delivered 27/04/2015)
• RCD (requested, delivered 03/03/2015)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_290_Danum_Valley

Data arrived from network transfer on 16/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Some corrupt webcam images were deleted: 141017_025820_000.jpg

Sensors:

• Fenix (requested)
• Leica FW LIDAR (?)
• Leica LIDAR (requested)
• RCD (requested)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

• 1220-1221
• 1159-1163 (also day 295)
• 1237-1243

May be three copies of 1239, logsheet says aborted.