Changes between Version 2 and Version 3 of Processing/Splitting

Aug 19, 2014 10:15:13 AM (5 years ago)

Added in instructions for how to split Fenix flightlines


  • Processing/Splitting

    v2 v3  
     1= Splitting Flightlines =
     3Some PIs will request that processed flight lines are split in to smaller lines, especially if the files generated are of a large size.
     5== Fenix Splitting ==
     7It is recommended that you do this after you have found the correct SCT offset values and are ready to generate the final files, as the splitting is done by aplcal when you submit the jobs to the grid. The splitting of the flight lines can be done by following this process:
     91. Find out the number of smaller segments the PI wants the flight lines split in to. This could be in the ticket or in emails. Ask someone if unsure.
     101. If it is not already there, append {{{cal_args_extra = -lines}}} to each flight line section of the config file. This tells aplcal which raw lines to process for this particular flight line section, using the {{{-lines}}} argument of aplcal.
     11   a. '''Note:''' If there is a global {{{cal_args_exrtra = }}} record in the config file (i.e. in the {{{[DEFAULT]}}} section, at the top of the file), this will be overridden by ones local to flight lines. Therefore, you will need to copy the arguments from the global section in to each flight line section to overcome this.
     121. Copy and paste each flight line section in the hyperspectral processing file <number of segments required> -1 times below the original flight line section so that the number of sections for each line is equal to the number of segments required by the PI.
     131. Divide the number of lines in each raw header file (the {{{lines}}} record in the raw header files (found in .../<root project directory>/hyperspectral/fenix) by the number of segments required and round it up/down to something sensible.
     141. After the {{{cal_args_extra = -lines}}}, put in the numbers of the raw flight lines to process.
     15   a. For the first one, this will be 1 (the first flight line) and the number you obtained in the previous step.
     16   a. If no overlap is required, the next section should be double the result, and so on. If an overlap is required, the first number used in the latter of the two should be <overlap in metres> less than the second number used in the section before it.
     171. Change the header of each flight line section so that the numbers in the section headers are sequential, and change the {{{line_id}}} and {{{line_number}}} fields to match this.
     181. Change the {{{raw_filebase}}} in each flight line section so that they point to the raw files. Do this by changing what's already there to {{{raw_filebase = %(fenix_filebase)s-<original line number>}}}. You can find this out by doing <new flight line number> / <number of sections required by PI>, and rounding it up.
     20Below is part of an example config file that was used for splitting hyperspectral data. In this example, each flight line was split in to three segments and used an overlap of 500m.
     24process_line = false
     25line_id = -1
     26line_number = 01
     27altitude = 2037
     28airspeed = 137
     29track = 102.0
     30raw_filebase = %(fenix_filebase)s-1
     31output_filebase = f%(julian_day)s%(sortie)s%(line_number)s
     32sct = 0.98
     33cal_args_extra = -lines 1 8000
     36process_line = false
     37line_id = -2
     38line_number = 02
     39altitude = 2037
     40airspeed = 137
     41track = 102.0
     42raw_filebase = %(fenix_filebase)s-1
     43output_filebase = f%(julian_day)s%(sortie)s%(line_number)s
     44sct = 0.98
     45cal_args_extra = -lines 7500 16000
     48process_line = false
     49line_id = -3
     50line_number = 03
     51altitude = 2037
     52airspeed = 137
     53track = 102.0
     54raw_filebase = %(fenix_filebase)s-1
     55output_filebase = f%(julian_day)s%(sortie)s%(line_number)s
     56sct = 0.98
     57cal_args_extra = -lines 15500 24557
     60process_line = true
     61line_id = -4
     62line_number = 04
     63altitude = 1922
     64airspeed = 141
     65track = 282.0
     66raw_filebase = %(fenix_filebase)s-2
     67output_filebase = f%(julian_day)s%(sortie)s%(line_number)s
     68sct = 0.96
     69nav2_args_extra = -force
     70cal_args_extra = -lines 1 8000
     73process_line = false
     74line_id = -5
     75line_number = 05
     76altitude = 1922
     77airspeed = 141
     78track = 282.0
     79raw_filebase = %(fenix_filebase)s-2
     80output_filebase = f%(julian_day)s%(sortie)s%(line_number)s
     81sct = 0.96
     82nav2_args_extra = -force
     83cal_args_extra = -lines 7500 16000
    187== Splitting LiDAR point clouds ==
    2 == Discrete LiDAR ==
     88=== Discrete LiDAR ===
    490In the case of purely discrete point clouds we can use lasmerge to produce equal divided files.
    14100Use -o to specify the number of seconds of overlap, this can be used with the airplanes ground speed to calculate overlap. For instance 7 seconds 140 knots will result in an overlap of 504 metres (7 seconds * 72 m/s). Give the script a fraction for the number of lines to divide the line into, I usually just give it 3.
    16 == Full Waveform ==
     102=== Full Waveform ===
    18104Full wave form data cannot use the script above. Unfortunately you will need to produce this using alspp and its fence feature.