Support: 25/Jul/08, Andrew Wilson, Alignment error in delivered data

[benj]

Initially raised at the end of July. Andrew had an issue whereby his calculated level 3 data did not seem to align with his vectors in ERDAS Imagine. Following some investigation, it seems that azexhdf produces GeoTiffs that report as being in the GRS80 ellipsoid (British National Grid normally relates to Airy). This is because modern BNG projection is now technically defined as being in relation to the ETRS89 datum (GRS80 ellipsoid) with a transform that results in the BNG co-ordinates being the same as they were when it was related to the OSGB36 datum (Airy ellipsoid), and the GeoTiff format doesn't cope with this very well.

However, for practical purposes it may be more useful to have azexhdf produce GeoTiffs reporting as Airy ellipsoid, since while this is technically incorrect now it makes no practical difference to the co-ordinates, and is less likely to confuse viewers like ERDAS Imagine.

That caused the main problem, though it also turned out that the wrong boresight numbers had been used to process this flight, so redelivery with correctly-processed data is necessary.

Email exchange follows:

Bill etal at PML,

I've just been helping Alasdair MacArthur do some of his Eagle processing, using Level 1b hdf files supplied to him from PML. The file in question is e102a021b.hdf.

I have azgcorred it using a 5m NextMap DEM with 1 metre output and using osgb02. In comparison with the Landline Plus vectors it required additional time and attitude offsets using -ut +0.70  and -ua 0.0 0.5 0.0     ie. 1/2 degree in roll. I have not seen this "attitude" problem with any of the Eagle data I have processed, the default sensor view angle vectors derived for the season working perfectly. The only problem I have had is the scan time offset ie. displacement along the flightline and occasionally a relative time offset between scan time and attitude time ie. the roll is not always perfectly aligned with the image roll distortion. As I have the level 1b IDS software I am able to apply any timing correction.

It would appear that azgcorr is missing the parameter option to apply a relative time offset of the attitude in relation to the scan time, although I recall from previous conversations with Bill that doing it at the azgcorr stage may not be the most appropriate place.

Any thoughts on a solution and time scale. Alasdair is a PhD student with lots of data to process.

If we aren't able to correct this error with azgcorr, I can see alot of users sending the data back for re-processing. A change to azgcorr would get PML off the hook for repeat processing.

Cheers,

Andrew

Hi Andrew

It's possible that there is a timing offset we missed in that flight - we obviously do our best to correct those but sometimes some slip through. In particular the scan position offset is easy to miss, because it can be corrected to look correct with the more common GPS timing offset. You know that already, I'm sure - sorry, I'm not trying to be patronising here. I'm actually still more inclined to blame incorrect boresight numbers - the pitch difference between what was used and what should have been was 0.48 degrees, which would have shifted the image along-track. It seems more likely to me that it would be a result of a known mistake that was made than that there was also an SCT offset present that was missed (not that I'm ruling that out, just that I'm inclined to go for the explanation that only requires one thing to go wrong). We're going to reprocess it anyway to fix the boresight, so we'll check the timing again in any case.

We do compare the final images to the vector data, and sometimes that does result in us going back and having another look at the timing. However, it's not uncommon for the vectors and the image to be a few pixels apart (often by differing amounts in different areas of the image), so it's possible for a mistake to get through that check. I think this is the first time we've had a user tell us that the timing was wrong, so it hasn't come up before, but I suspect our response would be (and is) "Sorry, we'll reprocess it for you" rather than advising the users to try and fix it themselves. As you say, the current azgcorr options don't really provide a facility for that, although I see from Bill's email that he's done some work in that direction already so possibly it'll be something that'll happen in future. 

Ben

Ben,

Alasdair wasn't interrupting me with this problem. We are working together on sorting out an atmospheric correction process for his Eagle data, so he was down from Edinburgh for a couple of days. While waiting to sort out a problem with envi FLAASH we tried out the azgcorr process. It was then that I discovered the problem with day 102.

However, after he had left I carried on and processed e191a021b.hdf and e241b021b.hdf with the inverclyde NextMap 5m DSM.

Although the 191 flight says (in the azgcorr listing) a centre pixel spatial resolution of 1.22m, the use of 1m output results in only 5% on the image being formed. With 1.5m out resolution about 80% is formed. It is only when I use 2m output does the whole image appear. It is rather a short image, so I don't know whether there was some mix up in the set scan rate integration time that would cause the big gaps when using approximately the correct spatial resolution for output. Even with this image, there is a clear mismatch of scan position of about 0.7 seconds (using the -ut 0.7 option in azgcorr). However, -ut seems to move the scan position, but fails to keep the attitude vectors aligned. This results in the scans being in about the right position along the scan line, but with completely useless attitude correction.

e24102 also requires about a -ut 0.7, but again the attitude is then incorrect. I tried using the the -us 50 option to move scan numbers, but whether I use 40, 50 or -50 nothing changes, at all, in the level 3 output image.

I'm not sure if I'm missing something obvious, but how are the users meant to correct for Eagle time offsets ?

Do you check any of the Level 3 outputs with an overlay of the Landline Plus vectors, as it should be fairly obvious that the images are not aligned, nor can be corrected by any available azgcorr command line option ?

As a processing team, do we have any options within the current setup to fix this. This is the first set of "user" (L1b hdf) data I have dealt with, so I'm worried this same level of mismatch is in all the other Eagle/Hawk data. I don't have any of Alasdair's Hawk data to check same, as suggested.

Cheers,

Andrew

PS: I believe Mike will have picked up Alasdair's hard disk last week.

PPS: I'm just ordering some large 1Tb hard disks, to back off all the data I have been sent, which will clear the backlog of the smaller hard disks I have accummulated. Unfortunately, our computer service cannot provide the necessary online RAID server space to hold all the CEH data.

PPPS: Can you provide me with a table for the view vector angles and lever arm offsets for Eagle and Hawk (6 parameters) and the time periods for which they are relevant. I want to make sure that my Level 1b processing batch jobs are using the correct settings for the different flight seasons.

Andrew, etc.

The adjustments in azgcorr to minimise residual errors in view vectors or nav ( the u* ones ) are correct for attitude but a fudge for time, whether complete epoch moves or attitude relative to position. This is because there is no reinterpolation done with the user time offsets. What I have looked at and suggested in the passed is to integrate some part of aznav functionality into the gcnav part of azgcorr. There are really two choices:

1. the easy way: to apply the time and other corrections, including changing the posn/attitude timing, to the CO vgroup nav vectors and then reinterpolating to get the per scan nav using the scan times.
 The downside to this is that if any time offsets are requested some scans at one or other end of the line will have no nav and have to be omitted from the final level 3.

2. on the other hand the nav vgroup vectors  generally have more nav at each end of the line so the above problem does not occur. It would look attractive to do this and I have already done a lot towards just embedding aznav into azgcorr; not particularly for applying user corrections but to make it easy to process third party data. If you are interested in this let me know and I will make available a test version. Initially  it will not write back the new CO vectors, just use them in azgcorr and then dump them! [ this version would be a FREE upgrade! ]

The downside here is that the updated CO vgroup would have to be written back to the level 1b file so that the final nav is preserved.

Horses for courses, I suppose!

Br Bill

Hi Andrew

It's possible that there is a timing offset we missed in that flight - 
we obviously do our best to correct those but sometimes some slip 
through. In particular the scan position offset is easy to miss, because 
it can be corrected to look correct with the more common GPS timing 
offset. You know that already, I'm sure - sorry, I'm not trying to be 
patronising here. I'm actually still more inclined to blame incorrect 
boresight numbers - the pitch difference between what was used and what 
should have been was 0.48 degrees, which would have shifted the image 
along-track. It seems more likely to me that it would be a result of a 
known mistake that was made than that there was also an SCT offset 
present that was missed (not that I'm ruling that out, just that I'm 
inclined to go for the explanation that only requires one thing to go 
wrong). We're going to reprocess it anyway to fix the boresight, so 
we'll check the timing again in any case.

We do compare the final images to the vector data, and sometimes that 
does result in us going back and having another look at the timing. 
However, it's not uncommon for the vectors and the image to be a few 
pixels apart (often by differing amounts in different areas of the 
image), so it's possible for a mistake to get through that check. I 
think this is the first time we've had a user tell us that the timing 
was wrong, so it hasn't come up before, but I suspect our response would 
be (and is) "Sorry, we'll reprocess it for you" rather than advising the 
users to try and fix it themselves. As you say, the current azgcorr 
options don't really provide a facility for that, although I see from 
Bill's email that he's done some work in that direction already so 
possibly it'll be something that'll happen in future.

Ben

Ben/Mike,

I think I have found "my" problem. Having gone back to square one and run e102a021b.hdf without any extra options (as I did first time), but this time loaded the image on top of yours, it works and lines up as we should have expected. I thought I must have just been going mad, but what happened orignally is the image I created I added to an ERDAS Imagine display with the Nextmap DSM and vectors already there. The difference is that your images come up in the ERDAS Imagine Image Info dialogue box as having a spheroid: GRS80 and a "not defined" datum. Mine came up with the ERDAS Imagine default OSGB 1936 datum that has an Airy spheroid.

To confirm, I took my image on top of yours and edited the Image Info to select the standard OSGB British National Grid projection that gives an Airy spheroid, saved and re-displayed. It comes out shifted away from your images and the vectors.

Could you confirm. If we are now meant to be using -mUK99 osgb02 in azgcorr we should be using a spheroid and datum of GRS80 in our Image Analysis packages ?? Presumably the ERDAS default standard BNG projection is no-longer correct ??

Phew !!

Andrew

Hi Andrew

Sorry for the delay in replying, it's taken us a little while to sort
out exactly what we think is going on. We've done some looking around and the best information we can find is at http://www.ordnancesurvey.co.uk/oswebsite/gps/information/coordinatesystemsinfo/guidecontents/guidea.html, which suggests that British National Grid projection (still) uses the OSGB36 datum, which in turn uses the Airy 1830 ellipsoid. This is certainly historically true, but Bill says that the most recent version of BNG uses the GRS80 ellipsoid. That's slightly confusing, because I'm not sure how they can just change a projection, and we've been unable to find any independent verification of that (as I said, the best information we can get from the OS website is that BNG still uses the Airy ellipsoid). I had a conversation with Bill on Monday where we discussed this, but I confess I came away still slightly unclear about how this all ties together. I've CC'd Bill, he may be able to shed some more light on this.

Bill has also said that the GeoTiff format is unable to support the British National Grid/OSGB36 projection with the latest transformations. This may be technically true, though the format is a rather murky one (Bill doesn't seem to like it much, and I don't blame him). It may be that a relatively accepted workaround for this now exists though, in that both ENVI and GRASS can produce GeoTiffs with the BNG/OSGB36 data included. We looked at them with GDALInfo and TiffInfo - I can send the results of that to you if you like. Again, Bill may be able to give us more information.

We tried some to-ing and fro-ing with variously-labelled files in ERDAS Imagine, ENVI and GRASS, and found (as I believe you did) that if the files were labelled as BNG/OSGB36 they lined up correctly with the vectors. This suggests to us that the output data is fine, they're just mis-labelled (probably as a result of the issue with GeoTiffs not supporting BNG properly). Imagine and ENVI do seem to have a tendency to re-project things without necessarily telling you, so it's possible we're wrong about that, but we think it's probably OK.

The upshot of all this is that we're not yet sure exactly what the solution to your problem is (whether we'll need an update to azgcorr/azexhdf or whether it's essentially correct and we'll just have to warn users), we're following this up with Bill to see what he thinks. We'll let you know when we've got a solution for you.

Ben