Support: 01/Oct/2007, Johanna Breyer, GB06-11

[mggr]

I am a PhD student at Aberystwyth university and received some CASI data
from you this June and while finally coming round to processing it, I
have come across the problems underlined below.
If you could give me some advice on this problem I would be very grateful.
Thank you and best wishes,

Johanna


The CASI imagery for lake Vyrnwy has been processed using the azgcorr
software version 4.7.9 (April 9 2007). But when the images are
processed they are flipped as shown in the attached image
(c219013a.jpg), for comparison see the attached screenshot sent with   the
original data (219-casi-mosaic.png). We have experimented with   the
command but cannot seem to fix the problem. The following command   (the
same as the readme file) was used to generate the attached
screenshot:

./bin/azgcorr -v -mUK99 ./misc/osgb02.cgrf -bl 5 3 2 -1 -p 2 2 -1 ./
lev1/c219011b.hdf  -3 ./lev3/c219013a.hdf

The DEM has been removed as we have a problem with it as the output   file
from the azgcorr process is blank when the DEM is included. We   are not
sure whether this is because of the flipping problem or
another separate problem.

In addition please find attached the exported details from the input and
output hdf files.

[mggr]

Replied suggesting:

• image appears flipped due to viewing a HDF rather than a GeoTIFF (not geolocated)
• blank output with DEM indicates DEM misses area or corrupt/misformatted DEM. Asked for azgcorr -v output and DEM info.

[mggr]

Pete Bunting (supervisor of Johannah) helped her a bit and continued to run into problems. It seems they were using a GeoTIFF for the DEM (which azgcorr should support, but didn't seem to do properly..). Advised they use an ASCII DEM for now.

Dear Mike. 

Thanks you for your previous reply it was useful, I am helping Johanna with some of her data processing and we are still having a problem when we include a DEM within the processing. The DEM is from nextmap and is in geotiff format and projected to the OS national grid it have been resampled to 2 m to match the CASI data. I have checked and it appears to cover the whole area covered by the CASI data but we still get blank images from the processing. I have copied the verbose output from the azgcorr program below in the hope that you have see what we are missing. But I am a bit confused by the DEM output:

Correction init  interpolation: BiCub  pixel size X: 2.00 Y: 2.00
DEM file: ../DEM/vrynwy_dem_2m_tif.tif
    limits SW: 0.0 0.0  NE: 0.0 0.0
    order (0): S-N cols: 0 rows: 0 grid inc: 10.0

** only 0  grid nodes from 784500 have values
** check if correct DEM file(s) have been used

DEM source: User flatfile
   name  : ../DEM/vrynwy_dem_2m_tif.tif
   limits  x:   294950 to  305400  y:   316500 to  323990
   (image) x:   295006 to  305342  y:   316552 to  323938
   cols: 1046  rows: 750  xinc: 10.0  yinc: 10.0
   height range  from: 622.5  to: 568.2
   NO geoid-spheroid correction applied

which initially seems to imply data the data is not being correctly read but then correctly reads the data in the next section include the projection and height values. 

Any information or thoughts would be useful and much appreciated. 

Best wishes,

Pete Bunting.

[pjb@localhost NERC_CASI]$ ./bin/azgcorr -v -mUK99 ./misc/osgb02.cgrf -eh ../DEM/vrynwy_dem_2m_tif.tif -p 2 2 -1 ./lev1/c219011b.hdf  -3 ./lev3/c219013a.hdf

-----------------------------------------------------------------------
azgcorr  -- ver: 4.7.9-lin Apr  9 2007   (C) Azimuth Systems UK 1996, 2007

data recorded in year: 2006  date: 07/08/2006
CCD type [1]: CASI scanner details and data

HDF level 1 input file: ./lev1/c219011b.hdf
CCD CApsfov table from hdf file with: 512 entries

Processing Spatial data


Navigation...
  navigation spheroid: WGS84  map projection spheroid: GRS80
  datum shift type: UK99  map projection: UKNG99

  site lat/lng start: 52.7427740 -3.4076599  end: 52.7990003 -3.5523020
       grid    start: 305068.1   317055.5   end: 295439.1   323510.4
       OS ref  start: SJ 506 1705          end: SH 9543 2351
  flight line grid azimuth: 303.8   direction to: West
  nav  index values from:     0  to:  6288
  scan index values from:     0  to:  6288
  navigation attitude items...
     Pitch  use status: file value unchanged
     Roll  use status: file value unchanged
     Heading  use status: file value unchanged

Pixel spacing stats... along line...
   centre     min: 1.69 max: 1.98 ave: 1.84
   port       min: 0.70 max: 4.57 ave: 2.00
   starboard  min: 0.80 max: 5.71 ave: 2.02
   across line centre (pix 256): 2.36

Site details...
  input lines to be processed: 6289
  corrected image pixels : 5169  lines: 3694
  limits from scans x: 295007.3 305340.1  y: 316553.2 323937.0
  pixel/line limits x: 295006.0 305342.0  y: 316552.0 323938.0

HDF data items to level 3 output file: ./lev3/c219013a.hdf

Scanner: CASI image (CAimage) pixels: 512 lines: 7649 bands: 15
  config: VEG
  view angles port: -26.3117 star: 26.7592

Correction init  interpolation: BiCub  pixel size X: 2.00 Y: 2.00
DEM file: ../DEM/vrynwy_dem_2m_tif.tif
    limits SW: 0.0 0.0  NE: 0.0 0.0
    order (0): S-N cols: 0 rows: 0 grid inc: 10.0

** only 0  grid nodes from 784500 have values
** check if correct DEM file(s) have been used

DEM source: User flatfile
   name  : ../DEM/vrynwy_dem_2m_tif.tif
   limits  x:   294950 to  305400  y:   316500 to  323990
   (image) x:   295006 to  305342  y:   316552 to  323938
   cols: 1046  rows: 750  xinc: 10.0  yinc: 10.0
   height range  from: 622.5  to: 568.2
   NO geoid-spheroid correction applied

HDF image access...

read hdf  bands: 15 pixels: 512 lines: 7649
  Image input HDF file: ./lev1/c219011b.hdf
  input  level1 image: CAimage   bands: 15 pix: 512 lines: 7649
  HDF output file: ./lev3/c219013a.hdfclear
  output level3 image: CAimage   bands: 15 pix: 5169 lines: 3694
Correcting bands:   1
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
2
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
3
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
4
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
5
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
6
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
7
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
8
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
9
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
10
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
11
    
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
12
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
13
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
14
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points
15
** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

** profile buffer exceeded at: 0 points

Correction rate: 1464  input lines per second

updating hdf

End of azgcorr run - files complete

[pjb@localhost NERC_CASI]$


On 2 Oct 2007, at 11:11, Johanna Breyer wrote:
>
> From: "Mike Grant" <mggr@pml.ac.uk>
> Date: 1 October 2007 18:59:42 BDT
> To: <removed, see internal contact details page for Johanna's email>
> Cc: <arsf-processing@pml.ac.uk>
> Subject: Re: FW: CASI processing
>
>
> Hi Johanna,
>
> Steve is unavailable at the moment so I'll try to help you :)  Please keep arsf-processing@pml.ac.uk in the cc list so he can see what's happening and in case someone else needs to take over for me - that email address is monitored by multiple people.
>
> Steve Groom forwarded:
>> The CASI imagery for lake Vyrnwy has been processed using the azgcorr
>> software version 4.7.9 (April 9 2007). But when the images are
>> processed they are flipped as shown in the attached image
>> (c219013a.jpg), for comparison see the attached screenshot sent with   the
>> original data (219-casi-mosaic.png). We have experimented with   the
>> command but cannot seem to fix the problem. The following command   (the
>> same as the readme file) was used to generate the attached
>> screenshot:
>
> Hmm.. I'd guess it might just be down to how you're viewing the images.  I'll guess you're using ENVI, though this may apply to other packages too.  Here's the explanation for what happens in ENVI.
>
> If you view one of the HDFs produced by azgcorr, ENVI doesn't read in the georeferencing information (there's no standard for that in the HDF format), so it displays the data as it's stored in the file (upside down).  You can check if this is happening by asking for the cursor location - if it displays coordinates in pixels, it's not georeferenced (should be in lat/long or map coordinates).
>
> If you process the level 3 HDF to a format that ENVI knows how to access the georeferencing information, it'll display it with north at the top of the screen.  For example, make a GeoTIFF with something like "../bin/azexhdf -h ./lev3/c219013a.hdf -G ./lev3/c219013a.tif", then view the GeoTIFF in ENVI.  Hopefully it'll look the right way up then.
>
> If I've guessed the problem wrong, let me know a few more details about what you're viewing with what program and I'll have another think :)
>
>> The DEM has been removed as we have a problem with it as the output   file
>> from the azgcorr process is blank when the DEM is included. We   are not
>> sure whether this is because of the flipping problem or
>> another separate problem.
>
> Normally if you get a blank output when running with a DEM, it means that your DEM doesn't properly cover the area of interest or isn't being read correctly due to being in the wrong format.  When the image lines fall far outside the coverage of the DEM, azgcorr can't correct for the height so just blacks those bits out.
>
> If you read through the reams of azgcorr output, there are two things of relevance you might find.  First, the DEM site limits will be shown - if these are wildly wrong, it's probably misreading the DEM.  The second thing to look out for is a warning that only some of the cells of the DEM overlap those of the image - if the site limits are correct, that'd imply that the DEM is ok but that it doesn't cover everything it needs to.
>
> To deal with the first problem, make sure your DEM is in a format that azgcorr is happy with - the manual talks a bit about this, but I'm happy to explain in more detail too :)
>
> To deal with the second problem, you'd need to know why the DEM wasn't covering the site.  This could be due to a coordinate conversion error (if it's in UTM and your image is in UK National Grid), getting the wrong NextMap tile (I do that a lot ;) ) or, if it's from LIDAR, that the swath wasn't wide enough to cover the area (extrapolate it in something like ENVI).
>
> If that doesn't make anything jump out at you, please email us the full output from the "azgcorr -v ..." line and some information about your DEM, and I'll see if we can spot the problem :)
>
> As a side issue, there's now a newer version (4.8.3) of azgcorr available which you can get from ftp://sw:nogwugAk7@ftp.nerc-arsf.ac.uk/ (clickable link).  The main differences are to do with processing LIDAR point clouds, so it shouldn't have any impact on your problems - just letting you know :)
>
> Hope that helped :)
>
> Cheers,
>
> Mike Grant.
> ARSF data processing support.
> --------------------------------------------------------
> Plymouth Marine Laboratory
> Prospect Place
> Plymouth
> PL1 3DH
>
> Website: www.pml.ac.uk
> Registered Charity No. 1091222
> Company No. 4178503
>
> PML is a member of the Plymouth Marine Sciences Partnership Website: www.pmsp.org.uk
>
> Please think before you print.
> --------------------------------------------------------
> This e-mail, its content and any file attachments are confidential.
> If you have received this e-mail in error please do not copy, disclose
> it to any third party or use the contents or attachments in any way.
> Please notify the sender by replying to this e-mail or e-mail
> forinfo@pml.ac.uk and then delete the email without making any copies
> or using it in any other way.
>
> The content of this message may contain personal views which are not
> the views of Plymouth Marine Laboratory unless specifically stated.
>
> Email transmission cannot be guaranteed to be secure or error free as
> information may be intercepted, corrupted, lost, destroyed, arrive
> late or incomplete or contain viruses. Plymouth Marine Laboratory
> accepts no liability for any loss or damage which may be caused by
> software viruses.
>
>

-----------------------------------------------------------
Peter Bunting; Lecturer
<Contact details removed, see internal contact details page>
------------------------------------------------------------

[mggr]

ASCII DEM appears to be working.

Acquired copy of the DEM - we should test with this to see what the problem was (ie. is azgcorr's geotiff handling unhappy?). See ~arsf/support/20071011-GB2006_11-PeteBuntingJohannahBreyer/

[mggr]

We got a nice email from Johanna saying everything was working well now and she was looking at atmospheric correction :)

Dear Mike,
I am writing just to let you know, that we have finally achieved a
really good geocorrection of the CASI data and are very pleased with it,
so thank you very much for your help, that really cracked the problems.
Peter and I are looking forward to hear what you are finding concerning
the DEM we sent you. I would also like to ask whether you have any
suggestions regarding the atmospheric correction of surface reflectance
of the data, i.e. what would you use customarily?
We are currently experimenting with both ENVI/Flaash and ATCOR 4 from
Rese and are confident of obtaining good results eventually, but are
interested in other people's views as well.
Thank you again and best wishes,
Johanna

I sent her a few links to atmospheric code and pointed her to Peter Land and Andrew Wilson for extra info. Also asked her if we could have any nice pics she gets out at the end, for showing off.

The DEM issue will probably need following up with Bill (warned her he's on holiday, so unlikely to contact him before then).

[mggr]

Sent an explanation of the DEM issue - turns out GeoTIFFs aren't supported now (manual needs an update).

Johanna Breyer wrote:
> Peter and I are looking forward to hear what you are finding concerning
> the DEM we sent you.

I finally got hold of an answer on this one - it turns out that the GeoTIFF support in azgcorr was removed after problems due to not being able to support the huge range of datums/georeferencing system in the GeoTIFF standard.  Only raw and ENVI-style DEMs are now supported.  So, your DEM was fine but was in a format that azgcorr can't read (and it didn't complain when it should have..).

However, the manual still has a reference to being able to use GeoTIFFs - we'll make sure that's removed in the next revision :)

Longer term, we're hoping to be able to add some additional checks to azgcorr such that it'll just stop rather than carrying on with something it can't use (rather an innovation in usability there!).