Changes between Initial Version and Version 1 of Procedures/CASI-reprocessing


Ignore:
Timestamp:
Oct 13, 2021, 3:08:06 PM (3 years ago)
Author:
asm
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/CASI-reprocessing

    v1 v1  
     1= CASI Reprocessed data =
     2
     3So you have to reprocess data... Let's see if we can help you out with this. Good luck!
     4
     5
     6== Acessing the Data ==
     7
     8First of all, if the data is not in our system, download the data from CEDA: https://data.ceda.ac.uk/neodc/arsf/
     9The password is here: https://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Passwords
     10
     11You will also need to download the log files and any documentation you can find.
     12
     13
     14== Create project directory and setup ==
     15Create a project directory where in appropriate location: </users/rsg/arsf/arsf_data/year/flight_data/country/project>
     16
     17Choose an appropiate project name <ProjCode-year-_jjjs_Site> and create the project directories as "arsf" user by using
     18`build_structure.py -p . `
     19Then change to user 'airborne' and create the missing directories. If the year is not found in folder structure, you can build_structure from another year and simply edit manually the directories or move them.
     20
     21Once the structure is created, copy the hdf files to a new directory called processing/hyperspetral/flightlines/level1b/hdf_files
     22
     23If there is any project info needed not found in the log. You might be able to get from the hdf files.
     24First of all, get the hdf code activated for you
     25`source ~utils/python_venvs/pyhdf/bin/activate`
     26
     27Now you can get extra info from hdf files for example like this:
     28`get_extra_hdf_params.py e153081b.hdf `
     29
     30You might need to edit the information for some of the scripts. One example is:
     31`--base="Almeria" --navsys="D.Davies" --weather="good" --projcode="WM06_13" --operator="S. J. Rob" --pi="R. Tee" --pilot="C. Joseph"`
     32
     33If the project is not in the database or the processing status page, it is better to include it (just follow the usual steps for unpacking a new project). Otherwise, many of the commands below will need that extra metadata information or might simply fail.
     34
     35
     36== Extracting navigation and setting up==
     37
     38
     39On the next step, you need to **create the nav files** by extracting the navigation information from the hdf files. Run the following command for each line
     40{{{
     41bash
     42source ~utils/python_venvs/pyhdf/bin/activate
     43for HDF in `find $LEV1_DIR -name *.hdf`
     44do
     45  extract_casi_atm_level1_data.sh -H $HDF -O
     46done
     47}}}
     48Move the files to the navigation post processed directrory and rename them. Please note that it is the original navigation rather than "_post_processed" but the scripts will be looking for that keyword.
     49
     50If you have extracted the navigation, then you will be able to automatically create a DEM model by specifying the BIL navigation files directory and running:
     51`create_apl_dem.py --aster -b ./processing/hyperspectral/flightlines/navigation/interpolated/post_processed/`
     52
     53And  you can create the specim config file by specifying all metadata information:
     54{{{
     55generate_apl_config.py --base="Sitia" --navsys="D.Davies" --weather="good" --projcode="MC04_15" --operator="S. J. Rob" --pi="R. Tee" --pilot="C. Joseph"  -y 2005 -j 130 --hdfdir ./processing/hyperspectral/flightlines/level1b/hdf_files
     56}}}
     57
     58
     59== Processing flightlines ==
     60
     61With the specim config file, you can easily run the processing for each line. First, edit the config file as any other airborne request to add the projection, casi_pixelsize, DEM path... and make sure all information is correct. Then, copy the casi fov files to the directory: `processing/hyperspectral/flightlines/sensor_FOV_vectors`.
     62
     63Now you can process each line by running the code:
     64`process_casi_atm.py      processing/hyperspectral/2005131.cfg  --sct 0.00 --gpt 0.00 -s casi -v processing/hyperspectral/flightlines/sensor_FOV_vectors/edited_to_use/casi_fov_vectors.bil      -O       -X`
     65
     66Remember to activate `pyhdf` before (see above).  Please note that it needs the flag `-O` for original and `-X` to skip the aplxml metadata step that will otherwise break. You will need to create the xml files manually at the end.
     67
     68For creating the xml files, you will need to match all flightlines with their respective log, hdf, level1b and igm and run a command for each one of them. The sample command will be:
     69{{{aplxml.py --meta_type=i --sensor='casi' \
     70--project_dir="/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/"     \
     71--line_id=-1 --config_file="/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/2005130_from_hdf.cfg" \
     72--lev1_file=/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/flightlines/level1b/c130011b.bil  \
     73--raw_file=/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/level1b/c130011b.hdf  \
     74--lev1_header=/data/nipigon1/scratch/arsf/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/flightlines/level1b/c130011b.bil.hdr \
     75--logfile="/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/logfiles/?_2005-130_casi_-1.o163231503411" \
     76--output=/data/nipigon1/scratch/arsf/2005/flight_data/greece/130_MC04_15/processing/delivery/MC04_15-2005-130/flightlines/line_information/c130011b.xml     \
     77--sct=0  --maxscanangle=0 --area Sitia --piemail unknown --piname "G Ferrier" --projsummary "-" --projobjective "-"  -projcode="MC04_15" \
     78--igm_header=/users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/flightlines/georeferencing/igm/c130011b_p_sct0.00_utm.igm.hdr 
     79}}}
     80
     81You can also create the xml file for the project information by simply running:
     82{{{
     83aplxml.py --meta_type p --project_dir /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15 --output /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/project_information/MC04_15-2005_130-project.xml --config_file /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/2005130_from_hdf.cfg --lev1_dir /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/delivery/MC04_15-2005-130/flightlines/level1b/ --igm_dir /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/flightlines/georeferencing/igm/ --area Sitia --piemail unknown --piname "G Ferrier" --projsummary "-" --projobjective "-" --project_code "MC04_15"
     84}}}
     85
     86If all steps were succesful and data looks good, you are ready to create a delivery.
     87
     88
     89
     90
     91
     92== Delivery creation ==
     93You should create the structure first
     94{{{
     95make_arsf_delivery.py --projectlocation <insert location and name> \
     96                      --deliverytype hyperspectral --steps STRUCTURE -c processing/hyperspectral/2005131_from_hdf.cfg
     97}}}
     98If the info for this project is not found in the database, you need to specify `--projectinfo 'year'=2005 'jday'=131 'projcode'=MC04 'sortie'=None `
     99                     
     100If happy, then run again with `--final`.
     101
     102Now, many steps will fail for CASI at one point or another. You should first check the other steps, run first:
     103{{{
     104make_arsf_delivery.py --projectlocation <insert location and name> \
     105                      --deliverytype hyperspectral --notsteps STRUCTURE
     106}}}
     107
     108Inspect the output, and all steps. You can start by moving or copying all the files manually and renaming them accordingly.
     109
     110Once done, you can run the SCREENSHOTS step by running:
     111{{{
     112createimage.py --input /users/rsg/arsf/arsf_data/2005/flight_data/greece/130_MC04_15/processing/hyperspectral/flightlines/georeferencing/mapped --output /data/nipigon1/scratch/arsf/2005/flight_data/greece/130_MC04_15/processing/delivery/MC04_15-2005-130/screenshots/ --verbosity 0 --matchonly ^c.*.bil$ --mosaic casi_mosaic --bands 4 3 1
     113}}}
     114
     115The project information should have been created in the previous section running aplxml. You will be able to undertake the rest of the steps simply by running:
     116{{{
     117make_arsf_delivery.py --projectlocation <insert location and name> \
     118                      --deliverytype hyperspectral --steps COPYDOCS ASCIIVIEW
     119                     
     120}}}
     121If everything went according to plan, the delivery should be nearly all done. You are likely to encounter new errors along the processing chain so pay special attention to all steps error messages.
     122
     123
     124=== Delivery Readme file ===
     125The scripts for creating the readme file will not work for CASI. The easiest way, is to copy another reprocessed CASI text file and edit. You can use as an example projects 130 and 131 from 2005. Edit the tex file and remove all references to the mask files and any other section that does not apply. Complete the data quality section as required, you can use the reprocessed data quality report