How to maintain the portal

The portal was written by PML (adbe then later knpa) to provide GRIMM data collected by ARSF to end users. It was also a goal to have a way of displaying the data (currently instigated as 3D plot) and a way of searching for data within given geo-ordinates (works but needs major improvements).

It is currently live and address is:
​http://arsf-dan.nerc.ac.uk/grimm/

To edit it, you need to ssh onto our web server as rsgadmin: (password on wiki)
ssh rsgadmin@www

The grimm files are located here:
~rsgadmin/arsf-dan.nerc.ac.uk/html/grimm

Errors from the site will be logged here:
~rsgadmin/arsf-dan.nerc.ac.uk/logs/

knpa's stuff:
~knpa/grimm/

Data (CSV and raw and output from dust monitor): ~knpa/grimm/data/

Ben Barrett has been giving feedback about the portal and would like to be kept in the loop:
benjamin.barratt@…

All my emails with Ben (and some with GRIMM, the company) are saved in a folder here: /users/rsg/knpa/grimm/grimm_emails/

Structure/Files

cgi-bin/query_grimm_db.cgi - python cgi script that queries the postgres database for a supplied area/time and returns result

data/ - contains CSV data files that are graphed and can be downloaded, one per flight day

data_list.php - page that displays the list of csv files that we have available

data_search.html - the page that houses the time/area search tool

images/ - contains images such as the NERC banner

index.html - initial page with links to csv data (and plotting), area/time search tool, and this wiki

lib/ - contains libraries used by other scripts

scatter.php - page that houses the graph and allows you to download the csv

scripts/scatter.js - javascript that generates the graph

scripts/cavasXpress.min.js - javascript library used by above

templates/ - css file for formatting the appearance of the html pages

The database

The area/time selection tool uses information from a postgres database.

This database holds the same information as the CSV files, which were read in with the following script: ~knpa/grimm/input_file.csv

The database is currently on knpa's machine (knpa to remember to change before he goes).
If for some reason I can't do it before I go, it's located on pmpc1319 at: /local1/data/scratch/pgsql/data/

It was created as the user 'postgres', not knpa.

The database is called grimmdb and can be accessed with psql: psql grimmdb

I wrote down what I needed to do to create the database and configure it. In case this is needed again (you may need the command to restart it if the server goes down for example), this information is here:
How to set up an configure a postgres database

Other scripts

Phil's script to create the CSV files in real time: phil_grimgps_opver.py

A script I wrote to extract "vertical profiles" from the data: grimm_extract_vertical_profiles.py

Things to be fixed/improved

Mark1/knpa suggestions:

• Improve speed of area/date search

• Return area/date search results in a way that can be easily downloaded

• Add 2010 and 2013/14 values to web portal.
  • 2010: For this we only a few flights over Balaton. Also, it was before Phil's script so they are are in the raw form. I've used dustMonitor to convert them (.../grimm/data/dust_monitor_output/) but there are two things stopping them going on the website
    1. They are in mean mass (micro-grams per micrometer) rather than being a straight particle count. We could ask Ben Barratt for a formula to convert the values.
    2. The CSV files are different from Phil's ones and would need to be-arranged appropriately.
  • 2013-14: these need to be downloaded from thelma still, then should be able to add as normal (if they've used Phil's script). If they are raw files then need to convert as above.

• 2d graph needed

• example format of time is wrong (: vs -)

• move database from knpa PC to a server

• option to save graphs (png?)

• get rid of minor marklines on axes?

• allow plots of different particle sizes on same graph - different colours?

• get data list page to look ok in non-chrome browser

• improve interactivity of graph (less lag)

Suggestions by Ben Barratt:

• Update so can group particle sizes into ultrafine (<1micron), fine (<2.5micron) and coarse (>2.5micron). A display showing the ratio would also be helpful.

• Try and use this to calculate the particle mass (Pm) value, which should be displayed on our portal (ask Ben B for this)

• allow you to get vertical profiles of areas where the plane is ascending or descending

• should cut out periods of takeoff and landing, or give these with a warning, because this will be very noisy data, with spikes. *note - this sounds like it will clash with above suggestion!*

• should display all of the different sizes at once instead of just three variables, could use different colours for each class.