Creation of DEMs from LIDAR point clouds

For help in creating DEMS from LIDAR point clouds acquired in 2009 onwards, from the Leica ALS II system, go here

Pre 2009 deliveries - the OPTECH LIDAR

This guide assumes you have a LIDAR dataset comprising a number of files containing LIDAR lines of the following format:

217755.001130 30563577.97 4513423.12 1293.23   178 30563577.96 4513423.12 1293.28   178
GPS_time(pulse?) ZZeasting[last_pulse] northing[last_pulse] height(m)[last_pulse] intensity[last_pulse] ZZeasting[first_pulse] northing[first_pulse] height(m)[first_pulse] intensity[first_pulse]

Handy tip for Linux users: strip off the UTM zone numbers with this command:
cat Str123.all | sed 's/^\( *[.0-9]* *\)..\([.0-9]* *[.0-9]* *[.0-9]* *[.0-9]* *\)..\([.0-9]* *[.0-9]* *[.0-9]* *[.0-9]*\)/\1\2\3/' > Str123-stripped.all

Automatic method

This method runs a script file that does all the steps from the manual method below. Use it unless for some reason you can't.

1. If you don't already have one, create a GRASS location that is appropriate for the area covered by the Lidar data.

2. Run the script file

   lidar2dem.sh -g grass_mapset_directory [-b base_name] [-m|-n] [-r|-o] [-h]
   

If you specify a base name then intermediate files (data, mask and histogram) will be created using the given base name with .xyz, .mask and .hist extensions respectively. These can then be re-used for subsequent lidar creation runs if desired (to recreate only one of these files, delete the relevant one and the others will be reused). The base name if given is also used for naming maps within GRASS - if no base name is specified then "lidar" is used for this. If you specify -m (default) then a GRASS mask file will be created and used (speeds up processing by interpolating less blank space), or -n specifies no mask. Specifying -r (default) results in any existing intermediate files with the correct name being re-used, or -o results in them being overwritten if present (ie all intermediate files are re-created from the data files. Specifying -h displays usage information.

Note also that it is recommended that you check the displayed upper and lower bounds against the displayed histogram to ensure that the automatically chosen values are sensible. If they are not and this causes a problem with the DEM then it will be necessary to reprocess using the manual method as below.

Manual method

1. Create a directory listing of the data files and save it to a file:

   \ls -l *.all.bz2 | awk '{print $8}' > dir_file
   

2. Merge files, strip off unnecessary UTM zone number and generate a GRASS mask by running generate_mask.py in the directory containing the lidar files:

   python generate_mask.py dir_file mask_file merged_file
   

...where dir_file is the file containing your directory listing, mask_file is the point file to be generated and used as a GRASS mask and merged_file is the merged and trimmed lidar data file. Note for ARSF internal use there is a convenience script in the path - use "generate_mask.sh dir_file mask_file merged_file"

3. Generate a histogram for the lidar data to determine min/max reasonable data values by running lidar_histogram.py:

   python lidar_histogram.py merged_file
   

Note for ARSF internal use there is again a convenience script in the path - use "lidar_histogram.sh merged_file"

This generates a 5x log histogram (ie it's a histogram of the base-10 log of the lidar values binned into 10m bins, multiplied by 5 to scale). If there are a small number of low or high values in the histogram then decide on a scaling cutoff. eg:

20: 
40: #################
50: #############################
60: #############################
70: ##############################
80: ##############################
90: ###############################
100: ###############################
110: ###############################
120: ################################
130: ################################
140: ###############################
150: #############################
160: ############################
170: ############################
180: ###########################
190: ###########################
200: ##########################
210: #######################
220: ###################
230: ##############
590: #####
600: ######
610: ######
620: ####

In the given example, there are a very few values between 20-30m at the bottom end, and a big gap at the top end before a small number of values above 590m (nothing between 240m and 590m). So it would be sensible to pick 30m as a minimum cutoff and 250m as a maximum. Note that because the histogram is on a log scale the difference in numbers of points at both ends is much larger than it looks at first glance. The script will display values that it thinks are appropriate cut-off values.

4. Check the LIDAR data projection in the header file. Open GRASS, select (or create) an a location in the appropriate projection and set the region appropriately for the dataset - geographic boundaries for the data are given by the script in the previous step, though you may wish to add a small amount of padding to this to ensure that the produced DEM covers the flight area.

5. Read in the mask file (assuming you want to use a mask)

   v.in.ascii input=mask_file output=mask_points format=point fs=, x=1 y=2 --overwrite
   

6. Generate the convex hull of the given points to use as a mask

   v.hull -a input=mask_points output=mask_poly --overwrite
   

7. Convert the vector convex hull to a raster map

   v.to.rast input=mask_poly output=raster_mask use=val value=1 --overwrite
   

8. Set the mask map to the calculated polygon

   r.mask -o input=raster_mask maskcats=*
   

9. Read in the trimmed lidar file using r.in.xyz. Note that for large areas this will use a LOT of memory - if the command refuses to start complaining of a lack of memory, use the percent argument and the command will automatically run in several passes - this will take longer, but use less memory on each pass.

   r.in.xyz input=<trimmed_lidar_file> output=<lidar_basemap> x=2 y=3 z=4 fs=" " zrange=<min_cutoff>,<max_cutoff> [percent=xx]
   

10. Interpolate gaps in the map using r.surf.idw

    r.surf.idw input=lidar_basemap output=lidar_interpolated --overwrite
    

11. Clear the GRASS mask

    r.mask -r input=anything
    

12. Reproject the dataset if necessary to match the projection of the flight data (should be the same, but conceivably won't be). See Creation of DEMs from SRTM 90m data for how to do this.

13. Generate an ASCII DEM as per Creation of DEMs from NextMap data:

    r.out.ascii input=<lidar_interp_map> output=lidar.dem null=0