= Archiving projects with NEODC =

=== Creating JSON files ===
First, create the JSON files for the delivered data. Those files must be archived later as well. Make sure all projects have been finalised go to the main project dir as '''airborne''' user and run:  
{{{
create_ceda_json.py -p .
}}}
The script will identify the different deliveries under delivery directory and files will be placed in under''' processing/json_files/'''''''. The type of delivery will be taken from the delivery name if it has the keywords: owl, hyperspectral, lidar or camera. Once the files have been created, have a look to the overview image and compare to the mosaic in the delivery to make sure everything is correct. Also open one JSON file in a text editor to check quickly all fields are correct. '''Specially check the''' '''ceda_path''' looks OK. If the project has been already archived, you will be able to get the link by opening  http://data.ceda.ac.uk/ followed by the ceda_path. If the project has not been archived, check that the path looks similar to others in the same period of time.

You can also specify the delivery directory by using {{{-d}}} and even run two different '-d' statements as well as you can choose a different output folder using {{{-o}}}. If you want to create a JSON file for each photograph file, then use {{{--camera_per_frame}}}. If at any time the overview image is not displaying correctly (not centred and because of that not displaying all data or labels) re-run the script with option {{{-i}}}. That will create an interactive plot that will allow you to check and show all data in the plot before exiting the script. 


=== Committing the JSON to the PostGIS database ===
If all JSON files look OK under processing/json_files we need that data in our database. If this is a new project you can upload all JSON files for the different deliveries by running:
{{{
json_into_postgis.py -p .
}}}

If  you want to only commit a specific type of JSON (camera, owl, lidar or hyperspectral) because other deliveries were already committed; you can do so by using the {{{--json_dir}}} argument insetad of {{{--project}}}. For example, if you want to add only the camera_json files just run:
{{{
json_into_postgis.py -j ./processing/camera_jsons/
}}}

In the case that the project has been reprocessed and new JSON files have been created, we do not want duplicates in the database. Running the script with the flag {{{--reprocessed}}} will delete the data in the sensor table (if data for that sensor were already present) and commit the new data. For example, if the camera has been redelivered and new JSON files have been created you can run:
{{{
json_into_postgis.py -j ./processing/camera_jsons/  --reprocessed
}}}
You can visualise the PostGIS database using QGIS by following this simple tutorial: [https://nerc-arf-dan.pml.ac.uk/trac/wiki/Processing/Visualise_PostGIS Visualise PostGIS] 

=== Archiving ===
This page documents the procedure to follow when sending data to NEODC.

 1. Choose a project:
  - A project is ready to be archived when all sensors have been delivered.
  - Check the ticket and make sure there is nothing on there which suggests something still needs to be done with the dataset (ask the processor if need be).
 1. Record in ticket that you are beginning to archive this flight.
 1. If there is a workspace version, move it into workspace/being_archived (pre-2011 only)
 1. Make sure deliveries have been finalised using 'finalise_delivered_data.sh' and use the current naming conversion of ProjectCode-Day-Sensor-Data.
 1. Prepare the repository version for archiving: 
   1. Run proj_tidy.sh to highlight any problems:
{{{
proj_tidy.sh -p <project directory> -c
}}}
     Check the output. Delete hidden('''not''' .htaccess or .htpasswd)/temporary/broken files. Fix incorrect file name formats and any other obvious errors. Everything in the delivery should be close to perfect, but don't worry too much about things in the main project.
   1. Make sure that the delivery sensor data is present and the raw data is present (incl. navigation). Run a quick eye over the rest of the deliveries.
   1. Remove unwanted large files. The project should have been cleaned up by the processor but often large files remain which are not needed. In particular, there are sometimes duplicates of data in processing/<sensor> which are included in delivery. Free up as much space as possible by deleting unwanted large files.  Don't delete anything in processing/kml_overview
   1. Add a copy of the relevant trac ticket(s) and zip; run '''as arsf''':
{{{
TICKETNUMBER=<ticket number of flight to be archived>
}}}
{{{
chmod u+w -R admin
mkdir -p admin/trac_ticket
pushd admin/trac_ticket
wget --recursive --level 1 --convert-links --html-extension http://nerc-arf-dan.pml.ac.uk/trac/ticket/$TICKETNUMBER
zip -r nerc-arf-dan.pml.ac.uk.zip nerc-arf-dan.pml.ac.uk
rm -fr nerc-arf-dan.pml.ac.uk
popd
chmod u-w -R admin
}}}
   1. Set permissions:
    1. Remove executable bit on all files (except the point cloud filter and the run[aceh] scripts):[[BR]]
       ** Note - if you are processing 2011 or later, you will need to run the below commands as both arsf and airborne **
{{{
find -type f -user `whoami` -not -wholename '*pt_cloud_filter*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
}}}
    1. Give everyone read permissions (and execute if it has user execute) for the current directory and below: 
{{{
find -user `whoami` -exec chmod a+rX {} \;
}}}
   1. If there are multiple deliveries for a sensor (apart from the APL reprocessing), then put all but the old versions in a subdirectory called previous_deliveries. Make sure the newest version is a complete delivery. Fill in missing data with hardlinks using `cp -nl` if necessary. Ask for help if not sure.
   1. Make sure the delivery directory is in the top level (otherwise use finalise_delivered_data.sh)
    1. If you move the delivery directory the KML overview links need to be updated to the new delivery folder location  
    1. Delete the broken symlinks and run make_kmloverview.py
    1. Check the links in the processing/kml_oveview folders or use the link on the wiki to open the google earth KML file and check the download links work and that there is data under each of the tabs
 1. Upload the data to NEODC using neodc-archive.py. This will upload the data using rsync (see http://www.ceda.ac.uk/help/archiving-with-ceda/sending-data-to-ceda/ for more details on archiving data at NEODC using rsync). If you are not running as airborne and haven't set up a password file you will need to enter your password (listed on the [https://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Passwords Passwords] page).
{{{
ceda-archive.py --year 2012 --jday 324 --sortie c --rsync
}}}
  - Note: If you don't need to upload all data (e.g., procesed only) run the command without the `--rsync` flag, this will create the folder structure and print the rsync commands to be run manually.
 1. If for any reason rsync fails you can also upload the data via FTP (note the general password, not the rsync one, is used) using:
{{{
lftp arsfdan@arrivals.ceda.ac.uk
> mirror -R -L /tmp/neodc_fEYiiE/ET12_14-2012_324_Boset_raw
}}}
 1. Notify NEODC the data has been uploaded (current contact is: wendy.garland@stfc.ac.uk. cc arsfinternal).
 1. Record in the ticket that it has been uploaded to NEODC and include the date.
 1. When NEODC confirm they have backed up the data:
  1. Note in ticket that it has been backed up by NEODC.
  1. Change status page entry to "archived" for relevant sensors (if pre-2011 HS then only do this if both original and reprocessed HS have been archived).
  1. If workspace version present, delete from being_archived.
  1. Add the flight in the appropriate format and in the appropriate position in ~arsf/archived_flights.txt. This is version controlled in ~arsf/live_git_repos/config_files so needs to be committed after being changed (there is no remote copy so no need to push).
  1. If all sensors have been archived (including reprocessing) then close the ticket. Otherwise note why the ticket is being left open.
 1. Note, if you don't get a conformation email from NEODC you can check projects which are out of embargo by seeing if the raw and processed data are available from http://browse.ceda.ac.uk/browse/neodc/arsf. Use the same username and password as for the CEDA FTP.