Splitting Flightlines
Some PIs will request that processed flight lines are split in to smaller lines, especially if the files generated are of a large size.
Fenix Splitting
It is recommended that you do this after you have found the correct SCT offset values and are ready to generate the final files, as the splitting is done by aplcal when you submit the jobs to the grid. The splitting of the flight lines can be done by following this process:
- Find out the number of smaller segments the PI wants the flight lines split in to. This could be in the ticket or in emails. Ask someone if unsure.
- If it is not already there, append cal_args_extra = -lines to each flight line section of the config file. This tells aplcal which raw lines to process for this particular flight line section, using the -lines argument of aplcal.
- Note: If there is a global cal_args_exrtra = record in the config file (i.e. in the [DEFAULT] section, at the top of the file), this will be overridden by ones local to flight lines. Therefore, you will need to copy the arguments from the global section in to each flight line section to overcome this.
- Copy and paste each flight line section in the hyperspectral processing file <number of segments required> -1 times below the original flight line section so that the number of sections for each line is equal to the number of segments required by the PI.
- Divide the number of lines in each raw header file (the lines record in the raw header files (found in .../<root project directory>/hyperspectral/fenix) by the number of segments required and round it up/down to something sensible.
- After the cal_args_extra = -lines, put in the numbers of the raw flight lines to process.
- For the first one, this will be 1 (the first flight line) and the number you obtained in the previous step.
- If no overlap is required, the next section should be double the result, and so on. If an overlap is required, the first number used in the latter of the two should be <overlap in metres> less than the second number used in the section before it.
- Change the header of each flight line section so that the numbers in the section headers are sequential, and change the line_id and line_number fields to match this.
- Change the raw_filebase in each flight line section so that they point to the raw files. Do this by changing what's already there to raw_filebase = %(fenix_filebase)s-<original line number>. You can find this out by doing <new flight line number> / <number of sections required by PI>, and rounding it up.
Below is part of an example config file that was used for splitting hyperspectral data. In this example, each flight line was split in to three segments and used an overlap of 500m.
[fenix_-1] process_line = false line_id = -1 line_number = 01 altitude = 2037 airspeed = 137 track = 102.0 raw_filebase = %(fenix_filebase)s-1 output_filebase = f%(julian_day)s%(sortie)s%(line_number)s sct = 0.98 cal_args_extra = -lines 1 8000 [fenix_-2] process_line = false line_id = -2 line_number = 02 altitude = 2037 airspeed = 137 track = 102.0 raw_filebase = %(fenix_filebase)s-1 output_filebase = f%(julian_day)s%(sortie)s%(line_number)s sct = 0.98 cal_args_extra = -lines 7500 16000 [fenix_-3] process_line = false line_id = -3 line_number = 03 altitude = 2037 airspeed = 137 track = 102.0 raw_filebase = %(fenix_filebase)s-1 output_filebase = f%(julian_day)s%(sortie)s%(line_number)s sct = 0.98 cal_args_extra = -lines 15500 24557 [fenix_-4] process_line = true line_id = -4 line_number = 04 altitude = 1922 airspeed = 141 track = 282.0 raw_filebase = %(fenix_filebase)s-2 output_filebase = f%(julian_day)s%(sortie)s%(line_number)s sct = 0.96 nav2_args_extra = -force cal_args_extra = -lines 1 8000 [fenix_-5] process_line = false line_id = -5 line_number = 05 altitude = 1922 airspeed = 141 track = 282.0 raw_filebase = %(fenix_filebase)s-2 output_filebase = f%(julian_day)s%(sortie)s%(line_number)s sct = 0.96 nav2_args_extra = -force cal_args_extra = -lines 7500 16000
Following is part of the raw header file used for the first flight line for the above example flight:
ENVI
description = { Copy of FENIX88-14-1.hdr with wavelengths replaced by wavelengths from /users/rsg/arsf/calibration/2014/fenix/fenix_201405.txt.}
samples = 384
lines = 24557
bands = 622
header offset = 0
file type = ENVI Standard
data type = 12
interleave = bil
byte order = 0
himg2 = {1, 384}
tint1 = 29.3000
tint2 = 25.9000
x start = 0
fps_set = 33.003
fps_qpf = 32.976
qpfTiming = {655.66, 744687.31}
GPS Start Time = UTC TIME: 10:47:32.0000
NavSync Timing = {0, 0}
GPS Stop Time = UTC TIME: 10:59:57.0000
sw_binning2 = {1, 1}
Splitting LiDAR point clouds
Discrete LiDAR
In the case of purely discrete point clouds we can use lasmerge to produce equal divided files.
A script exists to do this automatically:
split_las.py (-d INPUTDIR | -i INPUTFILE | -l INPUTLIST... ) -o OUTPUTDIR (--time time | --speed SPEED --distance DISTANCE )
This can be run on either a single file, a space separated list or a directory of las files. You can specify the number of seconds for overlap using --time (default is 1s aka 65ish metres), you can also specify the overlap in metres if you specify the plane speed too. If you dont want overlap use --time 0. This script will split full waveform and discrete LAS files.
lassplit Is a C++ program which will split LAS files based on a GPS time range.
lassplit -l las_input_file -o las_output_file -s gps_start_time -e gps_end_time
Full Waveform
Left this in as it is still useful.
It is worthwhile using the script above to divide a discrete line then using the x/y coordinates from each section for your division. This will give you a consistent split across the waveform data. If you have variable pitch/roll you will likely have to process each line seperately.
To give a fence in alspp go to filters > points on the ground. In the dialogue that displays click "selected projection (x,y,z)" then enter the min/max of your x/y coordinates for a square.
Alspp will not run on waveform data simultaneously with another alspp instance.
Seemingly at random some lines will not process with the fence options that every other line in the dataset will run under. This will result in an output of 22.6 k las files, you will need to change the fence size or in some cases create an entirely new alspp reg file to overcome this bug.
Once waveform data is completed you should test a couple from each segment in lashistoviewer or a similar waveform visualiser to check they are valid.