Back-up

This page describes how and where data is backed-up at ARSF-DAN.

The PML overnight back-up only covers home directories and does not make copies of anything on the arsf data storage disks. We therefore have our own scripts to backup things nightly to a different server.

Once flights are archived with NEODC, we omit them from the mirroring script. However, we still keep a local copy, on the same disk as the current data. It is also still covered by the rdiff script (see below).

Back-up locations

Back-up data should be linked from /users/rsg/arsf/backups/

There are two types of back-up:

1. Mirror of arsf_data:
   This is located at ~arsf/backups/arsf_data_mirror/. It is an exact copy of everything under ~arsf/arsf_data/ EXCEPT flights which have been archived with NEODC. If you accidently delete a file in arsf_data while processing, you can come here and retrieve a copy, as long as it was present for the overnight backup. The list of archived flights which are ommitted from the backup is at ~arsf/archived_flights.txt - this is updated as part of the archiving procedure.

2. Record of past changes:
   This is located at ~arsf/backups/rdiff_backup/. It holds a record of changes to small files (anything under 1MB I think). This way, we can recover changes to text files e.g. configs, made weeks or months ago. We can also find out the details of how something was processed by looking at the configs/logs for a given date. The rsync related command rdiff is used to create these. Each flight is in a different folder. The past changes are held in ~arsf/backups/rdiff_backup/<flight_dir>/rdiff-backup-data/. To recover the changes, please see documentation on rdiff.

Back-up Scripts

All backups are performed by the script at: ~arsf/usr/bin/repos_back_up_rsync_rdiff.sh

The cronjob that calls the above script is located at: /users/rsg/rsgcode/linux-i386/in_house/configs/cronjobs/crontab-arsf.gridmaster-SEC
Which writes to this log: ~arsf/backups/backup_log.txt

The crontab is currently set up on gridmaster3 to run at 4am every morning.

Potential problems

Occasionally there is a problem overnight and the rsync doesn't run until late. This can mean it is still running when the next rsync begins. This leads to a piling up of multiple instances of the script which just gets worse. It is good practice every now and then to log onto gridmaster3 as arsf and check that there aren't a stupid number of rsync/rdiff processes running (a few concurrent is normal). If so, just kill all instances you can find and let the cron start again as normal the following night. You could automatically check for this problem by modifying repos_back_up_rsync_rdiff.sh to check for rsync/rdiff processes and to fail if they are over a threshold.

rdiff-backup can also produce the following error "Found too many current_mirror incs!". A description of this error is available ​here, you can check using:

rdiff-backup -l GB08_19-2010_154a_London_day

There is a script to fix this problem by moving the duplicate files (/tools/backup/fix_rdiff_multiple_incs.py), if you think this needs running check with mark1 first.

Other problems can be found by checking the cron log above.