Imagico.de

blog

osmim_2017-01_980

January 5, 2017
by chris
0 comments

Additions to images for mapping

Just added a number of new images to the OSM images for mapping – here a few examples:

First is a Sentinel-2 image of the Central Alps in late September last year. This area is fully covered in high resolution images from other sources but many of them are at least partly not well suited for mapping due to snow or clouds. This image should be useful to update glacier extents in the area. There are also several other images of particular use for glacier mapping like the African glaciers which i featured here recently.

And there is an image of the Kerch Strait between the Sea of Azov and the Black Sea with the new bridge under construction there:

The newest image is of the Pacific side of the Panama Canal – an area which was cloud covered in the older Panama Canal image. This image was taken by the EO-1 satellite just a few days back.

The image was also taken at fairly low tidal water levels so the tidal flats at the coast are well visible.

EO1_980

December 18, 2016
by chris
0 comments

Early in the morning – the last days of EO-1

As most people know satellites generally have a limited life time. Space is a harsh environment, even for machines specifically designed to operate there. Satellites sometimes also fail because of construction and operation mistakes. But the most universal reason why satellites have a limited life span is because they run out of fuel.

If a satellite runs out of fuel its orbit altitude decays and it burns up in the atmosphere. Satellites in low earth orbit still fly in the upper parts of the earth atmosphere which are extremely thin but still produce some drag causing any satellite to gradually slow down and as a result lower its orbit. The International Space Station for example needs to raise its orbit several times per year because of that. Failure to do so would result in the ISS to burn up in the atmosphere within 1-2 years.

For earth observation satellites however this is not what happens when they run out out fuel – at least not initially. These satellites usually fly at a significantly higher altitude than the ISS and even without propulsion they usually remain flying for at least 30-50 years, sometimes significantly longer. How long this takes depends on the orbit altitude, the cross section of the satellite that produces drag relative to its mass and solar activity (which influences upper atmosphere density). The Envisat satellite i mentioned recently for example is expected to remain flying and not burn up in the atmosphere for about 150 years.

What happens with an earth observation satellite when fuel runs out is that it cannot maintain its sun synchronicity any more. And this happens much faster than orbital decay. The sun synchronous orbit of an earth observation satellite means its orbital plane rotates with the same speed as the earth rotates around its own axis but in the opposite direction so it flies with constant orientation of the orbit towards the sun. This happens because of the slightly non-spherical shape of the earth and by careful selection of the orbital parameters to make use of that. But this situation is unstable, there is no natural mechanism that maintains sun synchronicity so the satellite has to make adjustments to maintain this using its engine.

Landsat 7 is expected to run out of fuel next year. Here is a diagram from an USGS presentation illustrating what happens then.

What is shown on the y-axis is the local equator crossing time. As you can see this will move to earlier times quite rapidly and with increasing rate. During the time shown the orbit altitude will likely not change by more than a few kilometers.

There was another satellite in the same orbit as Landsat 7 that ran out of fuel in 2011: Earth Observing-1 or EO-1. I have shown images from EO-1 here on occasion in the past, its recordings are all available as open data just like Landsat imagery. EO-1 was a technology test platform evaluating new technologies for future earth observation satellites some of which have been realized on a larger scale in Landsat 8. EO-1 was started in 2000, about a year after Landsat 7 and originally planned to operate for one year. It is still running today which makes it the satellite most excessively exceeding its design life in history probably – an undead among satellites you could say. It was also – with a 10m resolution panchromatic band – the highest resolution open data satellite until the start of Sentinel-2.

Since EO-1 ran out of fuel more than five years ago it now has an equator crossing time early in the morning creating a fairly unique kind of images not available otherwise. Here an example of Mount Everest and the Rongbuk Glacier:

EO-1

Landsat 8

The EO-1 image on the left is from a day earlier but also more than two hours earlier (about 02:19 UTC compared to 04:42 UTC for Landsat). This view window gives fairly nice lighting conditions – as photographers know mid day light can often be relatively flat and boring while morning and evening situations are more likely to give interesting photo opportunities. Also relief is more articulated under such conditions. Here a few more examples, all of them from the second half of 2016.

Sierra Nevada

Appalachian Mountains

Tordrillo Mountains

Grand Canyon

Teton Range

Canyonlands

An early morning time window also means that at higher latitudes you get a better second late evening window during summer. You have than with Landsat too but much more limited and only available at very high latitudes. Here two examples from EO-1 from this year (from Kamchatka and Iceland).

Kamchatka

Iceland

The EO-1 ALI instrument from which all of the images here are derived pioneered many of the features we now have in Landsat 8 – like the shortwave blue band and the panchromatic band not extending into NIR. Its noise characteristics are not as good as with Landsat 8 – not surprising since it is 10 years older. In particular there is also some quite visible banding in the noise as can be seen in some of the images here. But it is still much better than Landsat 7. And the spectral characteristics (which are included in my satellite comparison chart) are in fact significantly better for true color visualizations than both Landsat 8 and Sentinel-2 due to broader red and green bands. You could actually say in this regard it represents the pinnacle in open data earth observation systems so far. I hope Landsat 10 will tie in with EO-1 in terms of visible band definitions but so far there does not seem to be a particular priority in that direction. It is hard to explain but working with EO-1 ALI colors is generally a real joy while tuning Landsat 8 or Sentinel-2 colors to get consistent, realistic and aesthetically pleasing results is often much more difficult.

Hawaii

Kilimanjaro

Northern Patagonian Ice Field

Coropuna, Peru

EO-1 is now scheduled to be deactivated in February – after nearly 17 years of operations. Despite being relatively lightweight (only about 500kg) it will remain flying at slowly decreasing altitude for many decades – see the following diagram from the report on decommissioning plans.

S3A_980c

December 13, 2016
by chris
0 comments

Sentinel-3 OLCI vs. MODIS/VIIRS – the overall tally

Based on my initial look at the Sentinel-3 data (see part 1, part 2 and part 3 for the details) here some comparitive assessments relative to the existing MODIS/VIIRS systems. As indicated in the title this is for OLCI, i.e. visible light and NIR only since the evaluation of SLSTR is incomplete due to errors in the geolocation data.

This is preliminary of course – based on the currently available data and my limited experience with that data. It covers a number of different fields i consider relevant for choosing a satellite data source. Which of the criteria listed are more important and which less of course depends on the use case.

# Topic Sentinel-3 OLCI MODIS VIIRS
1 Data point quality + o o
2 Data depth + o o
3 Resolution o o -
4 Image archive - ++ o
5 Revisit frequency -/o + ++
6 Temporal Coverage early Morning Morning + Afternoon Afternoon
7 Spatial Coverage - + +
8 Data packaging/usability + + o
9 Ease of data access o + +
10 Higher level products + +
11 License + ++ ++

More detailed notes about the individual categories:

  1. Data point quality refers to the quality of the individual data points (i.e. pixels) in the images, i.e. noise levels and radiometric accuracy. In terms of noise OLCI likely has an advantage over the >15 year old MODIS although as explained this will not really make much practical difference. Also both MODIS and VIIRS show a banding effect in images due to the way their scan system works which does not occur with OLCI.
  2. Data depth means the scope of information available for every point. Sentinel-3 OLCI leads here due to the additional bands. The narrow bands with somewhat suboptimal position w.r.t. visulization purposes reduce this but still a lead compared to MODIS and VIIRS which are not ideal either.
  3. Resolution – as discussed this is more or less a tie between OLCI and MODIS. VIIRS is clearly behind.
  4. Image archive refers to the size of the archive of past recordings that is available for use. Here MODIS has a huge lead of course.
  5. Revisit frequency means how frequently images are recorded at a given location. Apparently even with two satellites OLCI will not be better than MODIS with just a single satellite here. VIIRS leads with full daily coverage with even a single satellite.
  6. Temporal Coverage – is not really a rating category although you could say MODIS is the most versatile here with both morning and afternoon satellites. Since for Sentinel-3 the second satellite will cover the same time slot this will not change with Sentinel-3B.
  7. Spatial Coverage – this refers to what part of the Earth surface can be and is practically covered. Due to lack of south pole coverage Sentinel-3 is behind here. You also need to consider the relatively tight sun elevation cutoff of 10 degrees – limiting spatial or temporal coverage depending on how you look at it.
  8. Data packaging/usability – Both Sentinel-3 and MODIS have pros and cons here, no clear winner. VIIRS is somewhat behind due to relatively poor documentation and complex packaging and processing levels.
  9. Ease of data access – The Sentinel data access is – due to the mandatory use of the API and the need to register (except for the current so called pre-operation phase) – less convenient than the others.
  10. Higher level products – Currently Sentinel-3 data is only available as Level 1 data while the other sensors offer a broad selection of higher level products.
  11. License – all are open data but the vague attribution requirements for Sentinel-3 data are a possible issue for some applications.

And here for wrapping up three more views from Sentinel-3 OLCI – you can use these under CC-BY-SA conditions if you want:

water-reduced-980

December 12, 2016
by chris
0 comments

Reduced waterbody data on openstreetmapdata.com

I am happy to announce that reduced waterbody data for low zoom level rendering of OpenStreetMap based maps is now available on a regular basis on openstreetmapdata.com. This is produced using the methods i recently introduced here. You can find it on the waterbodies datasets page. Make sure you read the dataset description and the process description which contain important advise on how to use this.

These files can be used by anyone under the conditions of the OpenStreetMap license. Users should however consider supporting our service in financial or other form. Additions to the offered data like this as well as future availability of the free services in general depend on this support.

One additional thing to be aware of when using these files is that OpenStreetMap data always contains a significant amount of broken geometries which is sometimes visible in these files in form of missing features. Jochen maintains a page keeping track of broken polygons in OSM in general but we also generate a list with the errors affecting the waterbody rendering specifically. Many of these are just small water area no more significant that any other broken geometry in OSM but there are also quite a few really big polygons causing trouble. Anyone who is interested in improving data quality in OSM might consider working on this list. It is updated daily whenever we also update the waterbody data.

S3A_980b

December 11, 2016
by chris
0 comments

Sentinel-3 – a first look at the data, part 3

This post in the third and likely last part of a report on the recently released Sentinel-3 level 1 data from the OLCI and SLSTR instruments. The first part provided general context, the second part discussed the data distribution form and how this data can be processed and this third part will look more in depth into the quality of the data itself by comparing it to other open data satellite imagery available. Due to the problems with the SLSTR geolocation information explained in the second part this will only cover the OLCI instrument.

On data quality – OLCI

It is generally difficult to judge radiometric accuracy without reliable reference data. Noise characteristics of the visible bands appear to be pretty good – since recordings are limited to relatively high sun angles there is not really a good possibility to properly assess the dynamic range of course.

We remember the spectral bands chart from the first part – in the OLCI instrument there are essentially 10 spectral bands in the visible range, all of them very narrow. The usual convention will likely be to interpret band 3, 6 and 8 as blue, green and red. Bands 1 and 2 are near the blue edge of the visible range, band 7 is also well in the red range and can be used as a red band as well or in combination with band 8. Band 9 and 10 are near the long wavelength end of the visible range. Finally band 4 and 5 are in the blue-green area. The most problematic aspect for visualization purposes is the green band and its location right where red and green sensitivities of the human eye are about the same. Overall such narrow spectral sensitivities are not ideal for accurate color representation but it could be worse.

Resolution wise OLCI offers a 300m resolution at nadir in all bands. For comparison: MODIS and VIIRS provide different spatial resolutions in the red band and the green and blue bands. For MODIS that is 250m for red and 500m for grenn/blue. For VIIRS we get 375m for red and 750m for green/blue. Both also provide the higher resolution for an NIR band (and with VIIRS also for SWIR and thermal bands) so many analytic applications can also make use of higher resolution data.

The nominal resolutions tell only half of the story though. The wider the field of view of the satellite the stronger the dropoff in resolution is towards the edges. This means VIIRS has an additional disadvantage relative to MODIS if you consider the whole image average. The asymmetric view of Sentinel-3 OLCI means despite the more narrow recording swath the maximum viewing angle and therefore the resolution drop-off are comparable to that of MODIS.

Comparing with other satellites

So far this is all theory so here are some comparisons of images from the major open data satellite systems currently operating that provide visible light color images. It is not easy to find image sets where all of these record the same area on the same day, preferably all in the middle of the recording swath. I picked two areas and compromised by using MODIS images from one day earlier than the rest of the images which are all from the same day. The two areas are Lake Tchad and Northern Patagonia:

Images are based on calibrated radiances for all satellites. Normally i would have made this comparison based on reflectance values but apparently the necessary data to calculate reflectance values is currently not available for OLCI. This means differences in recording time and therefore illumination is not compensated for so the images differ also because of that. For reference here the recording times (in UTC) for all images shown.

Sensor Time Tchad Time Patagonia
VIIRS 12:30 19:00
MODIS (Terra) 09:45 14:55
Sentinel-3 08:56 14:14
Sentinel-2 09:38 14:54
Landsat 8 09:25 14:36

As you can see Sentinel-3 is generally the earliest due to the westward view while VIIRS with a noon viewing window is much later than the rest. Here is how the recording footprints look like, you can well see the different image sizes:

And here small area crops for all of these for comparison:

VIIRS Tchad

MODIS Tchad

OLCI Tchad

Sentinel-2 Tchad

Landsat 8 Tchad

VIIRS Patagonia

MODIS Patagonia

OLCI Patagonia

Sentinel-2 Patagonia

Landsat 8 Patagonia

Keep in mind both the differences in recording time and the different date for the MODIS image affect the results. What can still be observed however is:

  • that MODIS and Landsat 8 are fairly close in color calibration.
  • Sentinel-2 seems off relative to that – which i reported earlier. No sure way to say one of them is correct and the other is wrong though.
  • Sentinel-3 OLCI seems somewhere in between those in terms of base brightness but you need to take into account the earlier timeframe here. It does not show the same tint towards blue as the Sentinel-2 data despite the fact that the shortwave blue OLCI band 3 will likely feature significantly stronger atmosphere influence. To me this kind of indicates Sentinel-2 is the outlier here while the rest of the crowd is relatively well synchronized.
  • VIIRS is hard to compare due to the much later recording time but is probably also calibrated in combination with MODIS and Landsat by the USGS.
  • positional accuracy is generally not that great for the lower resolution images compared to the high resolution (Sentinel-2, Landsat) which are both very close so form a suitable reference.
  • resolution of Sentinel-3 OLCI and MODIS is fairly close. This is difficult to compare both due to the variability across the field of view and since MODIS provides higher resolution in the red channel than in the others – there are different possible approaches to use this to produce a high resolution full color image. Which is better also depends on how homogeneous colors are in the location you are looking at. Overall i think OLCI will usually have a slight advantage here overall for color images although for analytics involving mainly red and NIR like NDVI calculation MODIS probably offers measurably higher resolution. In any case the difference is fairly small compared to the difference between MODIS and VIIRS – which is not extremely big either near nadir but overall more significant due to the resolution falloff.
  • in the Patagonia sample area you can also see the advantage of the tilted view of Sentinel-3. MODIS, Sentinel-2 and Landsat all show sun glint on the lakes while OLCI does not. VIIRS also lacks sun glint in this area since the area is significantly off-center in the image to the east while the sun is slightly to the west.

Note for VIIRS and OLCI i mixed spectral bands for better approximation of visual colors.

Outlook

So far we only have the level 1 data. There are plans for higher level products to be made available in 2017 but these are fairly unspecific regarding the underlying methods. It is for example unclear if there will be a surface reflectance product of competitive quality. If we take MODIS as reference here – higher level MODIS products usually come with significant issues and limitations but their easy, uncomplicated and reliable availability makes them an attractive option if you are able to deal with these issues.

West African coast by Sentinel-3 OLCI

December 6, 2016
by chris
0 comments

Mobile cheese

I know this has been a long time coming but today ESA rolled out another change in their Sentinel-2 data distribution form. While the previous change was moving from multiple tile scenes to single tile packages – which i discussed previously resulting in significant performance problems as predicted this new change keeps the content of packages but changes the naming – both of the whole package and the internal file structure.

If you have read my Sentinel-2 data review you might remember that one of the first things i complained about there were the excessively blown up file names full of redundant data and information irrelevant to identify the files. I mentioned this because it is a nuisance when you work with the files but ultimately it is not a big deal – you just rename things into whatever form suits you when you ingest the data into your systems and then don’t have to worry about this any more. The idea that changing this now after more than a year of public data distribution is odd at best. Even more peculiar is the primary reason given for the change

The product naming (including the naming of folders and files inside the product structure) is compacted to overcome the 256 characters limitation on pathnames imposed by Windows platforms

Let me translate that: We now after more than a year of public data distribution change our data distribution form in a not backwards compatible way to cater users of a historic computer platform no more sold or even maintained by its creator that is so outdated that we did not even consider it and its limitations when we initially planned this 3-4 years ago.

Of course you could also simply say: 256 characters ought to be enough for anybody

This is how the change looks like: The old structure had package names like this:

S2A_OPER_PRD_MSIL1C_PDMC_20151230T202002_R008_V20151230T105153_20151230T105153.zip

and within that were data files like this:

S2A_OPER_PRD_MSIL1C_PDMC_20151230T202002_R008_V20151230T105153_20151230T105153.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20151230T162342_A002722_T31TFJ_N02.01/IMG_DATA/S2A_OPER_MSI_L1C_TL_SGS__20151230T162342_A002722_T31TFJ_B01.jp2

Now you get something like:

S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE.zip

and within:

S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE/GRANULE/L1C_T36JTT_A006424_20160914T081456/IMG_DATA/T36JTT_20160914T074612_B01.jp2

This shows just the main data files. The metadata and QA stuff is changed as well, many file names are now generic, that means they are identical for all packages – a bit like with the Sentinel-3 data, just that Sentinel-3 uses lower case file names while Sentinel-2 uses upper case file names.

There are also some quite sensible aspects in the change. For example the MGRS Tile ID is now in the package name. And the timestamps in the package name are in a different order, previously the processing time stamp was first while now the recording time steps is. This for example means when you sort the file names you get them in recording order rather than processing order which makes more sense.

The data distribution system continues to be very unreliable by the way so if you want to take this opportunity to download and look at some Sentinel-2 data you likely need quite a bit of patience.

Addition: The depth of obfuscation in the file format specifications is really impressive by the way. Looking for the actual meaning of the second time stamp in the package file name leads you to three different specifications. In the one that is currently distributed the second time stamp is apparently the datastrip sensing time but there are two other format variants where this is either

  • the package creation date or
  • the newest datastrip sensing time incremented by one second.

You can now really quite visualize what has happened here. Originally the creation date was meant to be used – this is at first mentioned everywhere in the specs. And then someone noticed that when processing the data in parallel the creation date is not necessarily unique…

S3A_980a

December 5, 2016
by chris
0 comments

Sentinel-3 – a first look at the data, part 2

This post in a continuation of the first look report on the recently released Sentinel-3 level 1 data from the OLCI and SLSTR instruments. The first part provided general context and discussion of the general data distribution form, this second part will look in more detail into the data itself.

The product data grid

I mentioned at the end of the first part that the form the imagery is distributed in is fairly strange. I will here try to explain in more detail what i mean. Here is how the OLCI data looks like for the visual range spectral bands. I combined the single package data shown previously with the next ‘scene’ generating a somewhat longer strip:

Sentinel-3 OLCI two image strip in product grid

What is strange about this is that this is not actually the view recorded by the satellite. For comparison here a MODIS image – same day, same area but due to the different orbits slightly further east. This is from the MOD02 processing level, that means radiomentrically calibrated radiances but without any geometric processing, i.e. this is what the satellite actually sees.

MODIS MOD02 image with geometry as recorded

You can observe the distortion at the edges of the swath – kind of like a fisheye photograph. This results primarily from the earth curvature, remember, these are wide angle cameras, in case of MODIS this covers a field of view of more than 90 degrees. With OLCI you would expect something similar, somewhat less distortion due to the more narrow field of view and asymmetric due to the tilted view. We don’t get this because the data is already re-sampled to a different coordinate system. That coordinate system is still a view oriented system that maps the image samples onto an ellipsoid based on the position of the satellite. In the ESA documentation this is called product grid or quasi-cartesian coordinate system.

This re-sampling is done without mixing samples just by moving and duplicating them. For OLCI this looks like this near the western edge of the recording swath:

Sentinel-3 OLCI pixel duplication near swath edge

For SLSTR things are more complicated due to a more complex scan geometry. Here this also results in samples being dropped (so called orphan pixels) when several original samples would occupy the same pixel in the product grid.

I have been trying to find an explanation on why this is done in the documentation and also thought about possible reasons. The only advantage you really have is that images in this grid are free of large scale distortion which is of course an advantage when you work with the data in that coordinate system. But normally you will use a different, standardized coordinate system that is not tied to the specific satellite orbit and when doing that this re-sampling is at best a minor nuisance, at worst the source of serious problems.

Getting the data out

So what do you need to get the data out of this strange view specific product grid? For this you need the supplementary data, in particular geo_coordinates.nc for OLCI and geodetic_*.nc for SLSTR. These files contain what is called geolocation arrays in GDAL terminology. You can use them to reproject the data into a standard coordinate system of your choosing. However the approach GDAL takes for geolocation array based reprojection is based on the assumption that there is a continuous mapping between the grids. This is not the case here and this leads to some artefacts and suboptimal results. There is a distinct lack of good open source solutions for this particular task despite this being a fairly common requirement so i have no real recommendation here.

What you get when you do this is something like the following. This is in UTM projection.

Sentinel-3 OLCI image reprojected to UTM

To help understanding what is going on here i also produced a version with only those pixels in the final grid with a source data sample within them being colored. Here two crops from this from near the nadir point and towards the western edge of the swath:

Point by point reprojection near nadir

Point by point reprojection near swath edge

This shows how sample density and therefore resolution differs between different parts of the image and how topography affects the sample distribution. You can also see that the image is combined from the views of several cameras and the edge interrupts the sample pattern a bit. In terms of color values these edges between the image strips are usually not visible by the way, you can sometimes see them on very uniform surfaces or in form of discontinuities in clouds.

Apart from longitude and latitude grids the files also contain an altitude grid. One can assume this is based on the same elevation data that is used to transform the view based product grid into the earth centered geographic coordinates. So lets have a look at this. For the Himalaya-Tibet area this looks like SRTM with fairly poor quality void fills – pretty boring, but all right.

Altitude grid in the Himalaya region

But in polar regions this is more interesting.

Altitude grid at the Antarctic peninsula

Altitude grid in Novaya Semlya

Not much more to say about this except: Come on, seriously?

So far we looked at the OLCI data. The SLSTR images are quite similar but there are of course separate geolocation arrays for the nadir and oblique images. However it looks like the geolocation data here is faulty. What you get when you reproject the SLSTR data based on the geodetic_*.nc is something like this:

Coordinate grid errors leading to incorrect resampling

The problem seems to manifest primarily in high relief areas so it is likely users only interested in ocean data don’t see it. There is also a separate low resolution ‘tie point’ based georeferencing data set and it is possible that this does not exhibit the same problem. I would also be happy to be proven wrong here of course – although this seems unlikely.

Original product grid (left), longitude component of the coordinate grid (center) and reprojected image (right)

The observed effect is much stronger in the oblique view imagery by the way.

SLSTR geolocation error in oblique view (left: product grid, right: reprojected)

This overall severely limits evaluation of the SLSTR data. What can be said so far is that otherwise the data itself looks pretty decent. There seems to be a smaller misalignment (a few hundred meters) between the VIS/NIR and the SWIR data as visible in the following examples in form of color fringes. It exists in both the nadir and the oblique view data sets. All of the SLSTR images in the product grid look kind of jaggy due to the way they are produced by moving and duplicating pixels, combined with the curved scan geometry of the sensor.

Sentinel-3 SLSTR example in false color infrared (left: nadir, right: oblique), click for full resolution

Sentinel-3 SLSTR false color example from the Alps (left: nadir, right: oblique)

The supplemental oblique view image is an interesting feature. Its primary purpose is most likely to facilitate better atmosphere characterization by comparing two views recorded with different paths through the atmosphere. Another reason is that just like with the laterally tilted view you avoid specular reflection of the sun itself. You might observe when comparing the two image that the oblique view is usually brighter. This has several reasons:

  • The path through the atmosphere is longer so more light is scattered.
  • Since it is a rear view it is recorded slightly later with a slighly higher sun position.
  • Since the rear view is pointed northwards on the day side it is looking at the sun lit flanks of mountains on the northern hemisphere – on the southern hemisphere it’s the other way round.

Other possible applications of the oblique view could be cloud detection and BRDF characterization of the surface. Note since the earth surface is much further away from the satellite in this configuration resolution of the oblique images in overall significantly lower.

So far we had an overall look at the data and how it is structured, the main characteristics and how it can be used. In the third part of this review i will make some comparisons with other image sources. Due to the mentioned issues with the SLSTR geolocation grids this will likely focus on the OLCI data only.

Here a few more sample images. The first one nicely illustrating the asymmetric view with sun glint on the far right near the maximum in the southern hemisphere summer in northeast Australia.

And here some more samples of various areas from around the world.

S3A_980

December 2, 2016
by chris
0 comments

Sentinel-3 – a first look at the data, part 1

When i recently wrote about the start of public distribution of Sentinel-3 data i indicated that i was going to write about this in more detail once SLSTR data is also available. This has happened several weeks ago already but it took somewhat longer to work through this than originally anticipated – although based on the experience with Sentinel-2 this is not really that astonishing.

This first part is going to be about the basics, the background of the Sentinel-3 instruments and the form the data is distributed in. More specific aspects of the data itself will follow in a second part.

On Sentinel-3

I already wrote a bit about the management side of Sentinel-3 when reporting on the data availability. Here some more remarks from the technical side. While Sentinel-1 and Sentinel-2 are both satellite systems with a single instrument Sentinel-3 features multiple different and independent sensors. Sentinel-3 is quite clearly a partial follow-up to the infamous Envisat project – something that is not too prominently advertised by ESA because it is a clear reminder of that corpse still lying around in the basement above out heads. Sentinel-1 could be understood to be a successor for the ASAR instrument on Envisat while most other Envisat sensors are reborn on Sentinel-3. I will here only cover the OLCI and SLSTR instruments which are related to the MERIS and AATSR systems on Envisat.

OLCI and SLSTR are included in my recent satellite comparison chart – here an excerpt from that in comparison with other similar systems:

All of these are on satellites in a sun synchronous polar orbit and record images continuously or at least for the dayside in resolutions between about 1000m and 250m. AVHRR is still running on four satellites but is to be preplaced in the future with VIIRS systems (which currently runs on one prototype satellite) GCOM-C SGLI is still in planning and not yet flying. Apart from spectral and spatial resolution the most important characteristic of such satellites is the time of day they record. Equator crossing time for Sentinel-3 is 10:00 in the morning which is closest to MODIS-Terra at about 10:30 (same is planned for GCOM-C). Since VIIRS is – both on Suomi NPP and on future JPSS satellites – going to cover a noon (13:30) time frame it could be that if MODIS-Terra ceases operations – which is not unlikely to happen with more than 16 years age – Sentinel-3 will be the only public source for morning time frame imagery in this resolution class.

That much for context. Now regarding the instruments: In short – OLCI is a visible light/NIR camera with 300m resolution in all spectral bands and SLSTR covers the longer wavelengths, starting from green, with 500m resolution (for visible light to SWIR) and 1000m resolution (for thermal IR). This somewhat peculiar division with overlapping spectral ranges is coming from the MERIS/AATSR heritage.

The most unusual aspect about the spectral characteristics is that OLCI features quite a large number of fairly narrow spectral bands at a relatively high spatial resolution of 300m.

OLCI records a 1270km wide swath. This means it takes three days to achieve full global coverage. For comparison MODIS covers 2330km and takes two days for full global coverage while VIIRS with 3000km coverage achieves full daily coverage. With the second Sentinel-3 satellite planned for next year OLCI will offer a combined coverage frequency comparable to MODIS. That being said OLCI is none the less very different because it looks slightly sideways. To cover 2330km from 705km altitude MODIS has a viewing angle from 55 degrees to both sides. OLCI on Sentinel-3 looks about 22 degrees to the east and 46 degrees to the west. In other words: it faces away from the morning sun coming from the east to avoid sun glint. This also means the average local time of the imagery is somewhat earlier than what is indicated by the 10:00 equator crossing time.

The SLSTR instrument is even more peculiar. It offers two separate views, one similar and overlapping with the OLCI view and 1400km wide, the other is more narrow (only 740km wide), not tilted to the west like the others but tilted backwards along the orbit of the satellite.

So what you get from OLCI and SLSTR together is three separate data sets:

  • The OLCI imagery, 1270km wide, view tilted 12.6 degrees to the west
  • The SLSTR nadir images, 1400km wide, likewise tilted to the west (so the ‘nadir’ is somewhat misleading) and fully overlapping with the OLCI coverage. The additional extent goes to the east so the SLSTR view is somewhat less tilted.
  • The SLSTR oblique/rear images, 740km wide, not tilted sideways

Here a visual example how this looks like (with SLSTR in classic false color IR):

Getting the data

But we are getting ahead of ourselves here. The publicly available Sentinel-3 data is available on the Sentinel-3 Pre-Operations Data Hub. What i am writing about here is the level 1 data currently released there. I am also not discussing the reduced resolution version of the OLCI data, just the full resolution OLCI and SLSTR data.

When you grab an OLCI data package from there it has a file name like this:

S3A_OL_1_EFR____20161120T044642_20161120T044942_20161120T063521_0179_011_133_2340_SVL_O_NR_002.zip

If you have read my Sentinel-2 review that probably looks painfully familiar. Excessively long, three time stamps, just in different order. First two time stamps are recording dates, the third one is processing. The footprints shown in the download interface like here:

are pretty accurate – at least after they fixed the obvious flaws when footprints crossed the edges of the maps after the first weeks. Don’t get confused when you frequently get every result twice (with different UUIDs but otherwise identical).

In the above file name i marked the components that are actually really useful in color. That is the date of acquisition, the relative orbit number and the along-track coordinate. The way the data is packaged is quite similar to Landsat (which is somewhat ironic since Sentinel-2 data is packaged much less like Landsat). The relative orbit number is like the Landsat path – just in order of acquisition rather than spatial order, the along-track coordinate is a bit like the Landsat row. Cuts along the track seem at roughly the same position usually and are made so that individual images have roughly square dimensions – but not precisely so numbers vary slightly from orbit to orbit. A number of other things are good to know when you look for data:

  • OLCI data is only recorded for the day side and only for areas with sun elevation above 10 degrees (or zenith angles of less than 80 degrees). This is quite restrictive, especially if you consider the early morning time frame recorded. Right now (end November) that limit is at about 60 degrees northern latitude. MODIS records to much lower sun positions (at least 86 degrees zenith angle AFAIK).
  • SLSTR is recorded continuously so you have descending and ascending orbit images. Only the thermal IR data is really useful for the night side of course.
  • The packages have no overlap in content.
  • Due to the view being tilted to the west in the descending orbit, i.e. to the right, images of both OLCI and SLSTR cover the north pole but not the south pole. The southern limit of OLCI coverage is at about 84.4 degrees south, SLSTR coverage ends at 85.8 degrees.

In the example shown the next package along the orbit is

S3A_OL_1_EFR____20161120T044942_20161120T045242_20161120T063600_0179_011_133_2520_SVL_O_NR_002.zip

- identical except for the times and the along-track coordinate.

Package content

When you unpack the zip you get a directory with the same base name and a ‘.SEN3′ at the end. When we look inside we might be positively surprised since we find only 30 files. For SLSTR that increases to 111 but still this is quite manageable compared to Sentinel-2. Of course – like with Sentinel-2, since the data is in compressed file formats packaging everything in a zip package is still very inefficient.

Image data is in NetCDF format. GDAL can read NetCDF and it can also read Sentinel-3 NetCDF but at least for the moment this is limited to basic reading of the data, any higher level functionality you need to take care of by hand. I assume GDAL developers will possibly in the future add specific support for the specific data structures of Sentinel-3 data but that might take some time.

What you have in the OLCI package is

  • One xml file xfdumanifest.xml with various metadata that might be useful, including for example also a footprint geometry.
  • 21 NetCDF files with names of the form Oa??_radiance.nc with ‘??’ being the channel number for the 21 OLCI spectral channels. This contains the actual image data.
  • Eight additional NetCDF files with various supplementary data.

It is great to have only 30 different files here so i am not complaining but the question is of course why they do not put all the data in one NetCDF file and save the need for a zip package? This is how MODIS data is distributed for example (in HDF format but otherwise same idea). I am just wondering…

In the SLSTR packages there is a bit more stuff but overall it is quite similar:

  • One xml file xfdumanifest.xml with metadata.
  • 34 NetCDF files for the image data of the 9+2 spectral bands with names of the form S?_radiance_[abc][no].nc or [SF]?_BT_i[no].nc with ‘?’ being the channel number. The ‘n’ or ‘o’ at the end indicates the nadir or oblique view as described above. ‘a’, ‘b’ or ‘c’ indicates different redundant sensors for the SWIR bands or a combination of them.
  • For each of the image data files there is a quality file with ‘quality’ instead of ‘radiance’ or ‘BT’ in the file name.
  • The other files are additional NetCDF packages with supplementary data – like with OLCI but different file names. Most of them are in different versions for the n/o and a/b/c/i variants.

Now if you pick the right ones from the 21 OLCI image data files, assemble them into an RGB image, which is fairly strait away with GDAL, you get – with some color balancing afterwards – something like this:

That might look pretty reasonable at the first glance but it is actually quite strange that the data comes in this form. More on that and other details on the data will come in the second part of this review.

While you wait here a few more image examples:


water-reduced-980

November 22, 2016
by chris
0 comments

Parting the waters

At the recent Hack Weekend in Karlsruhe i made some progress on a project i had been contemplating for quite some time and with further work in the last weeks first results can now be presented.

The main motivation for this was the problem of representing waterbodies in the OpenStreetMap standard style at low zoom levels. For a long time the standard OSM map has been showing the coastlines at all zoom levels based on data processed by Jochen and me but other water areas only on zoom level 6 and higher. The reason for this limitation is that rendering them at the lower zoom levels in the same way as at higher zoom levels would be very expensive regarding computing ressources.

Various solutions – or better: workarounds – have been proposed for this problem:

  • Using a different low detail data set for the low zoom levels – this is the usual lazy approach taken by many maps but with poor results regarding accuracy and cross-scale consistency. Commonly used data sets for this purpose like Natural Earth are often of exceptionally bad quality by todays standards.
  • Applying aggressive polygon size filtering, i.e. only rendering the largest geometries – an approach that is not advisable for OSM data because the way water areas are mapped in OSM and that would also be highly biased and distorting.
  • Tagging large or important waterbodies in a different way, either as coastline or with a newly created tag – of course manipulating the database to address some technical shortcomings of the rendering system is not a very good idea.

Generally speaking my techniques for geometric generalization of map data do already solve this problem but the subjective choices involved in this make using such an approach a sensitive matter. And a decent generalization of inland waterbodies is not really possible without a structural analysis of the river network which is a time-consuming endeavour that can not easily be performed on a daily basis. So the approach had to be less expensive and also more conservative and neutral in its results. The solution to this i now implemented has been in my mind for quite some time but until recently i never really found the time to fully work this out.

Example of waterbody rendering using the new technique

Looking back at things now makes me realize that what ultimately came out of this project is actually fairly peculiar from a technical perspective. This is likely useful for many people who render digital maps at coarse scales – but the fact that this approach makes sense also says something fairly profound about the way we currently render maps and the limits of these methods.

If this sounds confusing you can read up the whole background story – there you can also find links to the (for the moment still somewhat experimental) processed data.

The implementation of the technique introduced there is also available.

November 11, 2016
by chris
0 comments

Satellite comparison update

When writing about satellite images here i concentrate on open data imagery but i take the launch of a new commercial earth observation satellite by DigitalGlobe earlier today as opportunity to update my previously shown satellite comparison chart.

In addition to WorldView-4 i added the Terra Bella SkySat constellation. These satellites feature an interesting sensor concept and are meant to offer the currently unique ability to record video. There is however very little publicly known on operational plans for this system.

Also added is a column with the daily recording volume in square kilometers for the different systems. The most frequently advertised capability of satellites in addition to the spatial resolution is the revisit frequency which indicates how often a satellite can record images for any single point on the earth surface. A revisit interval of one day however does not mean the satellite system can record daily coverage of the whole earth surface. The recording volume describes what area can actually be covered on a daily basis. There are two variants: the potential recording volume (indicated in red) which is often a more or less theoretical value and the practical recording volume that is recorded in actual operations (in blue). In case of the commercial satellites both is based on claims by the operators obviously.

For the high resolution Open Data systems the numbers are determined based on average actual recordings. For Sentinel-2 this is a bit difficult so i based it on the specification of an average 14 minutes recording per orbit as specified in recent mission status reports. For the low resolution continuously recording systems numbers are – for compatibility – based on half orbit recordings although the longwave bands are of course recorded on the night side as well.

Generally marketing departments of satellite operators often seem to have a lot of fun twisting such numbers to make their systems look better. One thing to keep in mind here as reference: The earth land surface is about 150 million square kilometers. Landsat 8 records this almost fully (except for areas beyond 82.66 degrees latitude) every 16 days – but just barely. To do this it records approximately 25 million square kilometers every day or 400 million square kilometers in the 16 days global coverage interval. Now there are a couple of night side scenes included in that as well as quite a bit of ocean surface due to the width of the recording swath and the constraint of recording fixed WRS2 tiles. But it still illustrates that you need to record much more than just the 150 million square kilometers to actually cover the land surfaces in full, primarily due to the inevitable overlaps due to the orbit and recording footprint geometries.

S2A_R092_S05_20160802T075556_expose2.ann

October 28, 2016
by chris
0 comments

The African glaciers 2016

To those who are not so keen about satellite image related topics i am sorry for the recent focus on that matter – there will be other subjects again in the future. But for now i have here another imagery related post – about the glaciers of Africa.

Readers familiar with glaciers probably know there are three areas on the African continent with glaciation – all in the inner tropics. Glaciers in the tropics are very special since there is not such a clear seasonal pattern of winter snow accumulation and summer melt like at higher latitudes. All African glaciers have shown a significant retreat during the last century to a tiny fraction of their original size and will likely vanish completely within the next fifty years.

The least extensive glaciation exists on Mount Kenya

The Rwenzori Mountains at the border between Uganda and the Democratic Republic of the Congo feature somewhat more extensive glaciers due to the much wetter climate despite these mountains being the least tall of all three. Formerly glaciers were found on several peaks in the area but now they are mostly limited to the highest areas on Mount Stanley.

And finally the best known and tallest glaciated mountain in Africa is Mount Kilimanjaro. A hundred years ago most of the main caldera was still covered by ice while now there are only a few patches left. Due to the altitude glacier retreat on Kilimanjaro has very little to do with climate warming and more with decreasing amounts of snowfall and increasing sun exposure due to sunny weather.

All three images based on Copernicus Sentinel-2 data.

October 26, 2016
by chris
0 comments

I could have told you…

Short addition to the Sentinel-2 packaging and ESA API matter – three weeks ago i mentioned that

[...] Instead of browsing 200-300 packages per day you now have to deal with many thousands. This means the only feasible way to efficiently access Sentinel-2 data on a larger scale is now through automated tools.

And a few days ago:

Since ESA does not provide a bulk metadata download option you have to scrape the API for [obtaining bulk metadata].

Today the ESA announces that they think the recent API instability is largely caused by users attempting to retrieve extremely large numbers of results from queries and as a result they limit the maximum number of results returned by each query to 100. Three comments on that:

  • could have told them three weeks ago (maybe i should go into business as an oracle – although it really does not take a genius to predict that).
  • i doubt it will help much – it will create additional work for the programmers around the world to implement automatic followup queries to deal with the 100 rows limit – but in the end you will still have to serve out all the results, after all hardly anyone will do these queries for fun and there are now more than 160k entries to query – growing by about 4000-5000 every day (even though they now lag a week with publication of Sentinel-2 imagery). Systems like this which obviously do not scale to the level they are used at fail at the weakest link. But fixing or protecting this point of failure does not mean it will not fail at another place again. If the whole download framework goes down just because of a few unexpected but perfectly valid queries that is indication for a much more fundamental design problem.
  • even if it is so last century since today we all live in the cloud – offering bulk metadata download with a kind of daily/hourly diffs for updates might not be such a bad idea.

By the way (in case you wonder) – it was not me, when writing my scripts for creating the coverage analysis i was already working with a hundred rows limit – this was already the documented limit for the OData API anyway so it seemed appropriate to also use it in general.