December 13, 2016
by chris

Sentinel-3 OLCI vs. MODIS/VIIRS – the overall tally

Based on my initial look at the Sentinel-3 data (see part 1, part 2 and part 3 for the details) here some comparitive assessments relative to the existing MODIS/VIIRS systems. As indicated in the title this is for OLCI, i.e. visible light and NIR only since the evaluation of SLSTR is incomplete due to errors in the geolocation data.

This is preliminary of course – based on the currently available data and my limited experience with that data. It covers a number of different fields i consider relevant for choosing a satellite data source. Which of the criteria listed are more important and which less of course depends on the use case.

# Topic Sentinel-3 OLCI MODIS VIIRS
1 Data point quality + o o
2 Data depth + o o
3 Resolution o o -
4 Image archive - ++ o
5 Revisit frequency -/o + ++
6 Temporal Coverage early Morning Morning + Afternoon Afternoon
7 Spatial Coverage - + +
8 Data packaging/usability + + o
9 Ease of data access o + +
10 Higher level products + +
11 License + ++ ++

More detailed notes about the individual categories:

  1. Data point quality refers to the quality of the individual data points (i.e. pixels) in the images, i.e. noise levels and radiometric accuracy. In terms of noise OLCI likely has an advantage over the >15 year old MODIS although as explained this will not really make much practical difference. Also both MODIS and VIIRS show a banding effect in images due to the way their scan system works which does not occur with OLCI.
  2. Data depth means the scope of information available for every point. Sentinel-3 OLCI leads here due to the additional bands. The narrow bands with somewhat suboptimal position w.r.t. visulization purposes reduce this but still a lead compared to MODIS and VIIRS which are not ideal either.
  3. Resolution – as discussed this is more or less a tie between OLCI and MODIS. VIIRS is clearly behind.
  4. Image archive refers to the size of the archive of past recordings that is available for use. Here MODIS has a huge lead of course.
  5. Revisit frequency means how frequently images are recorded at a given location. Apparently even with two satellites OLCI will not be better than MODIS with just a single satellite here. VIIRS leads with full daily coverage with even a single satellite.
  6. Temporal Coverage – is not really a rating category although you could say MODIS is the most versatile here with both morning and afternoon satellites. Since for Sentinel-3 the second satellite will cover the same time slot this will not change with Sentinel-3B.
  7. Spatial Coverage – this refers to what part of the Earth surface can be and is practically covered. Due to lack of south pole coverage Sentinel-3 is behind here. You also need to consider the relatively tight sun elevation cutoff of 10 degrees – limiting spatial or temporal coverage depending on how you look at it.
  8. Data packaging/usability – Both Sentinel-3 and MODIS have pros and cons here, no clear winner. VIIRS is somewhat behind due to relatively poor documentation and complex packaging and processing levels.
  9. Ease of data access – The Sentinel data access is – due to the mandatory use of the API and the need to register (except for the current so called pre-operation phase) – less convenient than the others.
  10. Higher level products – Currently Sentinel-3 data is only available as Level 1 data while the other sensors offer a broad selection of higher level products.
  11. License – all are open data but the vague attribution requirements for Sentinel-3 data are a possible issue for some applications.

And here for wrapping up three more views from Sentinel-3 OLCI – you can use these under CC-BY-SA conditions if you want:


December 12, 2016
by chris

Reduced waterbody data on

I am happy to announce that reduced waterbody data for low zoom level rendering of OpenStreetMap based maps is now available on a regular basis on This is produced using the methods i recently introduced here. You can find it on the waterbodies datasets page. Make sure you read the dataset description and the process description which contain important advise on how to use this.

These files can be used by anyone under the conditions of the OpenStreetMap license. Users should however consider supporting our service in financial or other form. Additions to the offered data like this as well as future availability of the free services in general depend on this support.

One additional thing to be aware of when using these files is that OpenStreetMap data always contains a significant amount of broken geometries which is sometimes visible in these files in form of missing features. Jochen maintains a page keeping track of broken polygons in OSM in general but we also generate a list with the errors affecting the waterbody rendering specifically. Many of these are just small water area no more significant that any other broken geometry in OSM but there are also quite a few really big polygons causing trouble. Anyone who is interested in improving data quality in OSM might consider working on this list. It is updated daily whenever we also update the waterbody data.


December 11, 2016
by chris

Sentinel-3 – a first look at the data, part 3

This post in the third and likely last part of a report on the recently released Sentinel-3 level 1 data from the OLCI and SLSTR instruments. The first part provided general context, the second part discussed the data distribution form and how this data can be processed and this third part will look more in depth into the quality of the data itself by comparing it to other open data satellite imagery available. Due to the problems with the SLSTR geolocation information explained in the second part this will only cover the OLCI instrument.

On data quality – OLCI

It is generally difficult to judge radiometric accuracy without reliable reference data. Noise characteristics of the visible bands appear to be pretty good – since recordings are limited to relatively high sun angles there is not really a good possibility to properly assess the dynamic range of course.

We remember the spectral bands chart from the first part – in the OLCI instrument there are essentially 10 spectral bands in the visible range, all of them very narrow. The usual convention will likely be to interpret band 3, 6 and 8 as blue, green and red. Bands 1 and 2 are near the blue edge of the visible range, band 7 is also well in the red range and can be used as a red band as well or in combination with band 8. Band 9 and 10 are near the long wavelength end of the visible range. Finally band 4 and 5 are in the blue-green area. The most problematic aspect for visualization purposes is the green band and its location right where red and green sensitivities of the human eye are about the same. Overall such narrow spectral sensitivities are not ideal for accurate color representation but it could be worse.

Resolution wise OLCI offers a 300m resolution at nadir in all bands. For comparison: MODIS and VIIRS provide different spatial resolutions in the red band and the green and blue bands. For MODIS that is 250m for red and 500m for grenn/blue. For VIIRS we get 375m for red and 750m for green/blue. Both also provide the higher resolution for an NIR band (and with VIIRS also for SWIR and thermal bands) so many analytic applications can also make use of higher resolution data.

The nominal resolutions tell only half of the story though. The wider the field of view of the satellite the stronger the dropoff in resolution is towards the edges. This means VIIRS has an additional disadvantage relative to MODIS if you consider the whole image average. The asymmetric view of Sentinel-3 OLCI means despite the more narrow recording swath the maximum viewing angle and therefore the resolution drop-off are comparable to that of MODIS.

Comparing with other satellites

So far this is all theory so here are some comparisons of images from the major open data satellite systems currently operating that provide visible light color images. It is not easy to find image sets where all of these record the same area on the same day, preferably all in the middle of the recording swath. I picked two areas and compromised by using MODIS images from one day earlier than the rest of the images which are all from the same day. The two areas are Lake Tchad and Northern Patagonia:

Images are based on calibrated radiances for all satellites. Normally i would have made this comparison based on reflectance values but apparently the necessary data to calculate reflectance values is currently not available for OLCI. This means differences in recording time and therefore illumination is not compensated for so the images differ also because of that. For reference here the recording times (in UTC) for all images shown.

Sensor Time Tchad Time Patagonia
VIIRS 12:30 19:00
MODIS (Terra) 09:45 14:55
Sentinel-3 08:56 14:14
Sentinel-2 09:38 14:54
Landsat 8 09:25 14:36

As you can see Sentinel-3 is generally the earliest due to the westward view while VIIRS with a noon viewing window is much later than the rest. Here is how the recording footprints look like, you can well see the different image sizes:

And here small area crops for all of these for comparison:



OLCI Tchad

Sentinel-2 Tchad

Landsat 8 Tchad

VIIRS Patagonia

MODIS Patagonia

OLCI Patagonia

Sentinel-2 Patagonia

Landsat 8 Patagonia

Keep in mind both the differences in recording time and the different date for the MODIS image affect the results. What can still be observed however is:

  • that MODIS and Landsat 8 are fairly close in color calibration.
  • Sentinel-2 seems off relative to that – which i reported earlier. No sure way to say one of them is correct and the other is wrong though.
  • Sentinel-3 OLCI seems somewhere in between those in terms of base brightness but you need to take into account the earlier timeframe here. It does not show the same tint towards blue as the Sentinel-2 data despite the fact that the shortwave blue OLCI band 3 will likely feature significantly stronger atmosphere influence. To me this kind of indicates Sentinel-2 is the outlier here while the rest of the crowd is relatively well synchronized.
  • VIIRS is hard to compare due to the much later recording time but is probably also calibrated in combination with MODIS and Landsat by the USGS.
  • positional accuracy is generally not that great for the lower resolution images compared to the high resolution (Sentinel-2, Landsat) which are both very close so form a suitable reference.
  • resolution of Sentinel-3 OLCI and MODIS is fairly close. This is difficult to compare both due to the variability across the field of view and since MODIS provides higher resolution in the red channel than in the others – there are different possible approaches to use this to produce a high resolution full color image. Which is better also depends on how homogeneous colors are in the location you are looking at. Overall i think OLCI will usually have a slight advantage here overall for color images although for analytics involving mainly red and NIR like NDVI calculation MODIS probably offers measurably higher resolution. In any case the difference is fairly small compared to the difference between MODIS and VIIRS – which is not extremely big either near nadir but overall more significant due to the resolution falloff.
  • in the Patagonia sample area you can also see the advantage of the tilted view of Sentinel-3. MODIS, Sentinel-2 and Landsat all show sun glint on the lakes while OLCI does not. VIIRS also lacks sun glint in this area since the area is significantly off-center in the image to the east while the sun is slightly to the west.

Note for VIIRS and OLCI i mixed spectral bands for better approximation of visual colors.


So far we only have the level 1 data. There are plans for higher level products to be made available in 2017 but these are fairly unspecific regarding the underlying methods. It is for example unclear if there will be a surface reflectance product of competitive quality. If we take MODIS as reference here – higher level MODIS products usually come with significant issues and limitations but their easy, uncomplicated and reliable availability makes them an attractive option if you are able to deal with these issues.

West African coast by Sentinel-3 OLCI

December 6, 2016
by chris

Mobile cheese

I know this has been a long time coming but today ESA rolled out another change in their Sentinel-2 data distribution form. While the previous change was moving from multiple tile scenes to single tile packages – which i discussed previously resulting in significant performance problems as predicted this new change keeps the content of packages but changes the naming – both of the whole package and the internal file structure.

If you have read my Sentinel-2 data review you might remember that one of the first things i complained about there were the excessively blown up file names full of redundant data and information irrelevant to identify the files. I mentioned this because it is a nuisance when you work with the files but ultimately it is not a big deal – you just rename things into whatever form suits you when you ingest the data into your systems and then don’t have to worry about this any more. The idea that changing this now after more than a year of public data distribution is odd at best. Even more peculiar is the primary reason given for the change

The product naming (including the naming of folders and files inside the product structure) is compacted to overcome the 256 characters limitation on pathnames imposed by Windows platforms

Let me translate that: We now after more than a year of public data distribution change our data distribution form in a not backwards compatible way to cater users of a historic computer platform no more sold or even maintained by its creator that is so outdated that we did not even consider it and its limitations when we initially planned this 3-4 years ago.

Of course you could also simply say: 256 characters ought to be enough for anybody

This is how the change looks like: The old structure had package names like this:

and within that were data files like this:


Now you get something like:

and within:


This shows just the main data files. The metadata and QA stuff is changed as well, many file names are now generic, that means they are identical for all packages – a bit like with the Sentinel-3 data, just that Sentinel-3 uses lower case file names while Sentinel-2 uses upper case file names.

There are also some quite sensible aspects in the change. For example the MGRS Tile ID is now in the package name. And the timestamps in the package name are in a different order, previously the processing time stamp was first while now the recording time steps is. This for example means when you sort the file names you get them in recording order rather than processing order which makes more sense.

The data distribution system continues to be very unreliable by the way so if you want to take this opportunity to download and look at some Sentinel-2 data you likely need quite a bit of patience.

Addition: The depth of obfuscation in the file format specifications is really impressive by the way. Looking for the actual meaning of the second time stamp in the package file name leads you to three different specifications. In the one that is currently distributed the second time stamp is apparently the datastrip sensing time but there are two other format variants where this is either

  • the package creation date or
  • the newest datastrip sensing time incremented by one second.

You can now really quite visualize what has happened here. Originally the creation date was meant to be used – this is at first mentioned everywhere in the specs. And then someone noticed that when processing the data in parallel the creation date is not necessarily unique…


December 5, 2016
by chris

Sentinel-3 – a first look at the data, part 2

This post in a continuation of the first look report on the recently released Sentinel-3 level 1 data from the OLCI and SLSTR instruments. The first part provided general context and discussion of the general data distribution form, this second part will look in more detail into the data itself.

The product data grid

I mentioned at the end of the first part that the form the imagery is distributed in is fairly strange. I will here try to explain in more detail what i mean. Here is how the OLCI data looks like for the visual range spectral bands. I combined the single package data shown previously with the next ‘scene’ generating a somewhat longer strip:

Sentinel-3 OLCI two image strip in product grid

What is strange about this is that this is not actually the view recorded by the satellite. For comparison here a MODIS image – same day, same area but due to the different orbits slightly further east. This is from the MOD02 processing level, that means radiomentrically calibrated radiances but without any geometric processing, i.e. this is what the satellite actually sees.

MODIS MOD02 image with geometry as recorded

You can observe the distortion at the edges of the swath – kind of like a fisheye photograph. This results primarily from the earth curvature, remember, these are wide angle cameras, in case of MODIS this covers a field of view of more than 90 degrees. With OLCI you would expect something similar, somewhat less distortion due to the more narrow field of view and asymmetric due to the tilted view. We don’t get this because the data is already re-sampled to a different coordinate system. That coordinate system is still a view oriented system that maps the image samples onto an ellipsoid based on the position of the satellite. In the ESA documentation this is called product grid or quasi-cartesian coordinate system.

This re-sampling is done without mixing samples just by moving and duplicating them. For OLCI this looks like this near the western edge of the recording swath:

Sentinel-3 OLCI pixel duplication near swath edge

For SLSTR things are more complicated due to a more complex scan geometry. Here this also results in samples being dropped (so called orphan pixels) when several original samples would occupy the same pixel in the product grid.

I have been trying to find an explanation on why this is done in the documentation and also thought about possible reasons. The only advantage you really have is that images in this grid are free of large scale distortion which is of course an advantage when you work with the data in that coordinate system. But normally you will use a different, standardized coordinate system that is not tied to the specific satellite orbit and when doing that this re-sampling is at best a minor nuisance, at worst the source of serious problems.

Getting the data out

So what do you need to get the data out of this strange view specific product grid? For this you need the supplementary data, in particular for OLCI and geodetic_*.nc for SLSTR. These files contain what is called geolocation arrays in GDAL terminology. You can use them to reproject the data into a standard coordinate system of your choosing. However the approach GDAL takes for geolocation array based reprojection is based on the assumption that there is a continuous mapping between the grids. This is not the case here and this leads to some artefacts and suboptimal results. There is a distinct lack of good open source solutions for this particular task despite this being a fairly common requirement so i have no real recommendation here.

What you get when you do this is something like the following. This is in UTM projection.

Sentinel-3 OLCI image reprojected to UTM

To help understanding what is going on here i also produced a version with only those pixels in the final grid with a source data sample within them being colored. Here two crops from this from near the nadir point and towards the western edge of the swath:

Point by point reprojection near nadir

Point by point reprojection near swath edge

This shows how sample density and therefore resolution differs between different parts of the image and how topography affects the sample distribution. You can also see that the image is combined from the views of several cameras and the edge interrupts the sample pattern a bit. In terms of color values these edges between the image strips are usually not visible by the way, you can sometimes see them on very uniform surfaces or in form of discontinuities in clouds.

Apart from longitude and latitude grids the files also contain an altitude grid. One can assume this is based on the same elevation data that is used to transform the view based product grid into the earth centered geographic coordinates. So lets have a look at this. For the Himalaya-Tibet area this looks like SRTM with fairly poor quality void fills – pretty boring, but all right.

Altitude grid in the Himalaya region

But in polar regions this is more interesting.

Altitude grid at the Antarctic peninsula

Altitude grid in Novaya Semlya

Not much more to say about this except: Come on, seriously?

So far we looked at the OLCI data. The SLSTR images are quite similar but there are of course separate geolocation arrays for the nadir and oblique images. However it looks like the geolocation data here is faulty. What you get when you reproject the SLSTR data based on the geodetic_*.nc is something like this:

Coordinate grid errors leading to incorrect resampling

The problem seems to manifest primarily in high relief areas so it is likely users only interested in ocean data don’t see it. There is also a separate low resolution ‘tie point’ based georeferencing data set and it is possible that this does not exhibit the same problem. I would also be happy to be proven wrong here of course – although this seems unlikely.

Original product grid (left), longitude component of the coordinate grid (center) and reprojected image (right)

The observed effect is much stronger in the oblique view imagery by the way.

SLSTR geolocation error in oblique view (left: product grid, right: reprojected)

This overall severely limits evaluation of the SLSTR data. What can be said so far is that otherwise the data itself looks pretty decent. There seems to be a smaller misalignment (a few hundred meters) between the VIS/NIR and the SWIR data as visible in the following examples in form of color fringes. It exists in both the nadir and the oblique view data sets. All of the SLSTR images in the product grid look kind of jaggy due to the way they are produced by moving and duplicating pixels, combined with the curved scan geometry of the sensor.

Sentinel-3 SLSTR example in false color infrared (left: nadir, right: oblique), click for full resolution

Sentinel-3 SLSTR false color example from the Alps (left: nadir, right: oblique)

The supplemental oblique view image is an interesting feature. Its primary purpose is most likely to facilitate better atmosphere characterization by comparing two views recorded with different paths through the atmosphere. Another reason is that just like with the laterally tilted view you avoid specular reflection of the sun itself. You might observe when comparing the two image that the oblique view is usually brighter. This has several reasons:

  • The path through the atmosphere is longer so more light is scattered.
  • Since it is a rear view it is recorded slightly later with a slighly higher sun position.
  • Since the rear view is pointed northwards on the day side it is looking at the sun lit flanks of mountains on the northern hemisphere – on the southern hemisphere it’s the other way round.

Other possible applications of the oblique view could be cloud detection and BRDF characterization of the surface. Note since the earth surface is much further away from the satellite in this configuration resolution of the oblique images in overall significantly lower.

So far we had an overall look at the data and how it is structured, the main characteristics and how it can be used. In the third part of this review i will make some comparisons with other image sources. Due to the mentioned issues with the SLSTR geolocation grids this will likely focus on the OLCI data only.

Here a few more sample images. The first one nicely illustrating the asymmetric view with sun glint on the far right near the maximum in the southern hemisphere summer in northeast Australia.

And here some more samples of various areas from around the world.


December 2, 2016
by chris

Sentinel-3 – a first look at the data, part 1

When i recently wrote about the start of public distribution of Sentinel-3 data i indicated that i was going to write about this in more detail once SLSTR data is also available. This has happened several weeks ago already but it took somewhat longer to work through this than originally anticipated – although based on the experience with Sentinel-2 this is not really that astonishing.

This first part is going to be about the basics, the background of the Sentinel-3 instruments and the form the data is distributed in. More specific aspects of the data itself will follow in a second part.

On Sentinel-3

I already wrote a bit about the management side of Sentinel-3 when reporting on the data availability. Here some more remarks from the technical side. While Sentinel-1 and Sentinel-2 are both satellite systems with a single instrument Sentinel-3 features multiple different and independent sensors. Sentinel-3 is quite clearly a partial follow-up to the infamous Envisat project – something that is not too prominently advertised by ESA because it is a clear reminder of that corpse still lying around in the basement above out heads. Sentinel-1 could be understood to be a successor for the ASAR instrument on Envisat while most other Envisat sensors are reborn on Sentinel-3. I will here only cover the OLCI and SLSTR instruments which are related to the MERIS and AATSR systems on Envisat.

OLCI and SLSTR are included in my recent satellite comparison chart – here an excerpt from that in comparison with other similar systems:

All of these are on satellites in a sun synchronous polar orbit and record images continuously or at least for the dayside in resolutions between about 1000m and 250m. AVHRR is still running on four satellites but is to be preplaced in the future with VIIRS systems (which currently runs on one prototype satellite) GCOM-C SGLI is still in planning and not yet flying. Apart from spectral and spatial resolution the most important characteristic of such satellites is the time of day they record. Equator crossing time for Sentinel-3 is 10:00 in the morning which is closest to MODIS-Terra at about 10:30 (same is planned for GCOM-C). Since VIIRS is – both on Suomi NPP and on future JPSS satellites – going to cover a noon (13:30) time frame it could be that if MODIS-Terra ceases operations – which is not unlikely to happen with more than 16 years age – Sentinel-3 will be the only public source for morning time frame imagery in this resolution class.

That much for context. Now regarding the instruments: In short – OLCI is a visible light/NIR camera with 300m resolution in all spectral bands and SLSTR covers the longer wavelengths, starting from green, with 500m resolution (for visible light to SWIR) and 1000m resolution (for thermal IR). This somewhat peculiar division with overlapping spectral ranges is coming from the MERIS/AATSR heritage.

The most unusual aspect about the spectral characteristics is that OLCI features quite a large number of fairly narrow spectral bands at a relatively high spatial resolution of 300m.

OLCI records a 1270km wide swath. This means it takes three days to achieve full global coverage. For comparison MODIS covers 2330km and takes two days for full global coverage while VIIRS with 3000km coverage achieves full daily coverage. With the second Sentinel-3 satellite planned for next year OLCI will offer a combined coverage frequency comparable to MODIS. That being said OLCI is none the less very different because it looks slightly sideways. To cover 2330km from 705km altitude MODIS has a viewing angle from 55 degrees to both sides. OLCI on Sentinel-3 looks about 22 degrees to the east and 46 degrees to the west. In other words: it faces away from the morning sun coming from the east to avoid sun glint. This also means the average local time of the imagery is somewhat earlier than what is indicated by the 10:00 equator crossing time.

The SLSTR instrument is even more peculiar. It offers two separate views, one similar and overlapping with the OLCI view and 1400km wide, the other is more narrow (only 740km wide), not tilted to the west like the others but tilted backwards along the orbit of the satellite.

So what you get from OLCI and SLSTR together is three separate data sets:

  • The OLCI imagery, 1270km wide, view tilted 12.6 degrees to the west
  • The SLSTR nadir images, 1400km wide, likewise tilted to the west (so the ‘nadir’ is somewhat misleading) and fully overlapping with the OLCI coverage. The additional extent goes to the east so the SLSTR view is somewhat less tilted.
  • The SLSTR oblique/rear images, 740km wide, not tilted sideways

Here a visual example how this looks like (with SLSTR in classic false color IR):

Getting the data

But we are getting ahead of ourselves here. The publicly available Sentinel-3 data is available on the Sentinel-3 Pre-Operations Data Hub. What i am writing about here is the level 1 data currently released there. I am also not discussing the reduced resolution version of the OLCI data, just the full resolution OLCI and SLSTR data.

When you grab an OLCI data package from there it has a file name like this:

If you have read my Sentinel-2 review that probably looks painfully familiar. Excessively long, three time stamps, just in different order. First two time stamps are recording dates, the third one is processing. The footprints shown in the download interface like here:

are pretty accurate – at least after they fixed the obvious flaws when footprints crossed the edges of the maps after the first weeks. Don’t get confused when you frequently get every result twice (with different UUIDs but otherwise identical).

In the above file name i marked the components that are actually really useful in color. That is the date of acquisition, the relative orbit number and the along-track coordinate. The way the data is packaged is quite similar to Landsat (which is somewhat ironic since Sentinel-2 data is packaged much less like Landsat). The relative orbit number is like the Landsat path – just in order of acquisition rather than spatial order, the along-track coordinate is a bit like the Landsat row. Cuts along the track seem at roughly the same position usually and are made so that individual images have roughly square dimensions – but not precisely so numbers vary slightly from orbit to orbit. A number of other things are good to know when you look for data:

  • OLCI data is only recorded for the day side and only for areas with sun elevation above 10 degrees (or zenith angles of less than 80 degrees). This is quite restrictive, especially if you consider the early morning time frame recorded. Right now (end November) that limit is at about 60 degrees northern latitude. MODIS records to much lower sun positions (at least 86 degrees zenith angle AFAIK).
  • SLSTR is recorded continuously so you have descending and ascending orbit images. Only the thermal IR data is really useful for the night side of course.
  • The packages have no overlap in content.
  • Due to the view being tilted to the west in the descending orbit, i.e. to the right, images of both OLCI and SLSTR cover the north pole but not the south pole. The southern limit of OLCI coverage is at about 84.4 degrees south, SLSTR coverage ends at 85.8 degrees.

In the example shown the next package along the orbit is

- identical except for the times and the along-track coordinate.

Package content

When you unpack the zip you get a directory with the same base name and a ‘.SEN3′ at the end. When we look inside we might be positively surprised since we find only 30 files. For SLSTR that increases to 111 but still this is quite manageable compared to Sentinel-2. Of course – like with Sentinel-2, since the data is in compressed file formats packaging everything in a zip package is still very inefficient.

Image data is in NetCDF format. GDAL can read NetCDF and it can also read Sentinel-3 NetCDF but at least for the moment this is limited to basic reading of the data, any higher level functionality you need to take care of by hand. I assume GDAL developers will possibly in the future add specific support for the specific data structures of Sentinel-3 data but that might take some time.

What you have in the OLCI package is

  • One xml file xfdumanifest.xml with various metadata that might be useful, including for example also a footprint geometry.
  • 21 NetCDF files with names of the form Oa?? with ‘??’ being the channel number for the 21 OLCI spectral channels. This contains the actual image data.
  • Eight additional NetCDF files with various supplementary data.

It is great to have only 30 different files here so i am not complaining but the question is of course why they do not put all the data in one NetCDF file and save the need for a zip package? This is how MODIS data is distributed for example (in HDF format but otherwise same idea). I am just wondering…

In the SLSTR packages there is a bit more stuff but overall it is quite similar:

  • One xml file xfdumanifest.xml with metadata.
  • 34 NetCDF files for the image data of the 9+2 spectral bands with names of the form S?_radiance_[abc][no].nc or [SF]?_BT_i[no].nc with ‘?’ being the channel number. The ‘n’ or ‘o’ at the end indicates the nadir or oblique view as described above. ‘a’, ‘b’ or ‘c’ indicates different redundant sensors for the SWIR bands or a combination of them.
  • For each of the image data files there is a quality file with ‘quality’ instead of ‘radiance’ or ‘BT’ in the file name.
  • The other files are additional NetCDF packages with supplementary data – like with OLCI but different file names. Most of them are in different versions for the n/o and a/b/c/i variants.

Now if you pick the right ones from the 21 OLCI image data files, assemble them into an RGB image, which is fairly strait away with GDAL, you get – with some color balancing afterwards – something like this:

That might look pretty reasonable at the first glance but it is actually quite strange that the data comes in this form. More on that and other details on the data will come in the second part of this review.

While you wait here a few more image examples:


November 22, 2016
by chris

Parting the waters

At the recent Hack Weekend in Karlsruhe i made some progress on a project i had been contemplating for quite some time and with further work in the last weeks first results can now be presented.

The main motivation for this was the problem of representing waterbodies in the OpenStreetMap standard style at low zoom levels. For a long time the standard OSM map has been showing the coastlines at all zoom levels based on data processed by Jochen and me but other water areas only on zoom level 6 and higher. The reason for this limitation is that rendering them at the lower zoom levels in the same way as at higher zoom levels would be very expensive regarding computing ressources.

Various solutions – or better: workarounds – have been proposed for this problem:

  • Using a different low detail data set for the low zoom levels – this is the usual lazy approach taken by many maps but with poor results regarding accuracy and cross-scale consistency. Commonly used data sets for this purpose like Natural Earth are often of exceptionally bad quality by todays standards.
  • Applying aggressive polygon size filtering, i.e. only rendering the largest geometries – an approach that is not advisable for OSM data because the way water areas are mapped in OSM and that would also be highly biased and distorting.
  • Tagging large or important waterbodies in a different way, either as coastline or with a newly created tag – of course manipulating the database to address some technical shortcomings of the rendering system is not a very good idea.

Generally speaking my techniques for geometric generalization of map data do already solve this problem but the subjective choices involved in this make using such an approach a sensitive matter. And a decent generalization of inland waterbodies is not really possible without a structural analysis of the river network which is a time-consuming endeavour that can not easily be performed on a daily basis. So the approach had to be less expensive and also more conservative and neutral in its results. The solution to this i now implemented has been in my mind for quite some time but until recently i never really found the time to fully work this out.

Example of waterbody rendering using the new technique

Looking back at things now makes me realize that what ultimately came out of this project is actually fairly peculiar from a technical perspective. This is likely useful for many people who render digital maps at coarse scales – but the fact that this approach makes sense also says something fairly profound about the way we currently render maps and the limits of these methods.

If this sounds confusing you can read up the whole background story – there you can also find links to the (for the moment still somewhat experimental) processed data.

The implementation of the technique introduced there is also available.

November 11, 2016
by chris

Satellite comparison update

When writing about satellite images here i concentrate on open data imagery but i take the launch of a new commercial earth observation satellite by DigitalGlobe earlier today as opportunity to update my previously shown satellite comparison chart.

In addition to WorldView-4 i added the Terra Bella SkySat constellation. These satellites feature an interesting sensor concept and are meant to offer the currently unique ability to record video. There is however very little publicly known on operational plans for this system.

Also added is a column with the daily recording volume in square kilometers for the different systems. The most frequently advertised capability of satellites in addition to the spatial resolution is the revisit frequency which indicates how often a satellite can record images for any single point on the earth surface. A revisit interval of one day however does not mean the satellite system can record daily coverage of the whole earth surface. The recording volume describes what area can actually be covered on a daily basis. There are two variants: the potential recording volume (indicated in red) which is often a more or less theoretical value and the practical recording volume that is recorded in actual operations (in blue). In case of the commercial satellites both is based on claims by the operators obviously.

For the high resolution Open Data systems the numbers are determined based on average actual recordings. For Sentinel-2 this is a bit difficult so i based it on the specification of an average 14 minutes recording per orbit as specified in recent mission status reports. For the low resolution continuously recording systems numbers are – for compatibility – based on half orbit recordings although the longwave bands are of course recorded on the night side as well.

Generally marketing departments of satellite operators often seem to have a lot of fun twisting such numbers to make their systems look better. One thing to keep in mind here as reference: The earth land surface is about 150 million square kilometers. Landsat 8 records this almost fully (except for areas beyond 82.66 degrees latitude) every 16 days – but just barely. To do this it records approximately 25 million square kilometers every day or 400 million square kilometers in the 16 days global coverage interval. Now there are a couple of night side scenes included in that as well as quite a bit of ocean surface due to the width of the recording swath and the constraint of recording fixed WRS2 tiles. But it still illustrates that you need to record much more than just the 150 million square kilometers to actually cover the land surfaces in full, primarily due to the inevitable overlaps due to the orbit and recording footprint geometries.


October 28, 2016
by chris

The African glaciers 2016

To those who are not so keen about satellite image related topics i am sorry for the recent focus on that matter – there will be other subjects again in the future. But for now i have here another imagery related post – about the glaciers of Africa.

Readers familiar with glaciers probably know there are three areas on the African continent with glaciation – all in the inner tropics. Glaciers in the tropics are very special since there is not such a clear seasonal pattern of winter snow accumulation and summer melt like at higher latitudes. All African glaciers have shown a significant retreat during the last century to a tiny fraction of their original size and will likely vanish completely within the next fifty years.

The least extensive glaciation exists on Mount Kenya

The Rwenzori Mountains at the border between Uganda and the Democratic Republic of the Congo feature somewhat more extensive glaciers due to the much wetter climate despite these mountains being the least tall of all three. Formerly glaciers were found on several peaks in the area but now they are mostly limited to the highest areas on Mount Stanley.

And finally the best known and tallest glaciated mountain in Africa is Mount Kilimanjaro. A hundred years ago most of the main caldera was still covered by ice while now there are only a few patches left. Due to the altitude glacier retreat on Kilimanjaro has very little to do with climate warming and more with decreasing amounts of snowfall and increasing sun exposure due to sunny weather.

All three images based on Copernicus Sentinel-2 data.

October 26, 2016
by chris

I could have told you…

Short addition to the Sentinel-2 packaging and ESA API matter – three weeks ago i mentioned that

[...] Instead of browsing 200-300 packages per day you now have to deal with many thousands. This means the only feasible way to efficiently access Sentinel-2 data on a larger scale is now through automated tools.

And a few days ago:

Since ESA does not provide a bulk metadata download option you have to scrape the API for [obtaining bulk metadata].

Today the ESA announces that they think the recent API instability is largely caused by users attempting to retrieve extremely large numbers of results from queries and as a result they limit the maximum number of results returned by each query to 100. Three comments on that:

  • could have told them three weeks ago (maybe i should go into business as an oracle – although it really does not take a genius to predict that).
  • i doubt it will help much – it will create additional work for the programmers around the world to implement automatic followup queries to deal with the 100 rows limit – but in the end you will still have to serve out all the results, after all hardly anyone will do these queries for fun and there are now more than 160k entries to query – growing by about 4000-5000 every day (even though they now lag a week with publication of Sentinel-2 imagery). Systems like this which obviously do not scale to the level they are used at fail at the weakest link. But fixing or protecting this point of failure does not mean it will not fail at another place again. If the whole download framework goes down just because of a few unexpected but perfectly valid queries that is indication for a much more fundamental design problem.
  • even if it is so last century since today we all live in the cloud – offering bulk metadata download with a kind of daily/hourly diffs for updates might not be such a bad idea.

By the way (in case you wonder) – it was not me, when writing my scripts for creating the coverage analysis i was already working with a hundred rows limit – this was already the documented limit for the OData API anyway so it seemed appropriate to also use it in general.


October 22, 2016
by chris

Sentinel-2 – the first year

Sentinel-2 is an earth observation satellite mission producing open data imagery, the first satellite of which was launched in June last year and for which the data started to become available in November last year. The Sentinel-2 design is quite similar to Landsat and with a slightly higher spatial resolution of 10m it produces the highest resolution open data satellite imagery available at the moment.

This is to be a review of the image coverage generated by Sentinel-2 during the first year of operation. But for reference i will start with Landsat.


Landsat 8 has now been operating for more than three years and as indicated before it is essentially recording a relatively bias free global coverage of the land surfaces at a 16 day interval now. To illustrate here a plot of last years day side (descending orbit) acquisitions. This is generated by plotting the WRS2 footprints in a color representing the number of scenes available. This is fairly easy to do using the bulk metadata packages made available by the USGS.

The timeframe is not chosen arbitrarily, a mid October cut includes exactly one full northern hemisphere and southern hemisphere summer season.

As you can see for low latitudes the map is quite uniformly colored indicating 21-23 acquisitions on most major land areas, corresponding to the 16 day interval. At high latitude not all possible slots are recorded but as the denser lines already hint this does not necessarily mean less frequent image coverage. To better show the actual image recording frequency here a different visualization indicating the per-pixel image count for all pixels that include land areas.

Here you can see that the high latitudes are actually more frequently covered in most cases – which is not a bad choice since cloud incidence here is often higher than in the subtropics.

A few interesting observations can be made here. First you can see the off-nadir recordings in northern Greenland and the Antarctic extending the coverage and reducing the area not covered at all, which is drawn in blue. This is the central Antarctic of course but there are also two pixels further north that are blue:

  • Rockall – which is not really large enough to be recorded on Landsat imagery in any meaningful way.
  • Iony Island – which is a clear omission since other smaller islands elsewhere are specifically covered by Landsat.

Here this island on a Sentinel-2 image:

What Landsat does not see – Iony Island by Sentinel-2

Apart from these the lower latitudes are recorded at every opportunity so there is not much that can be improved here. At the higher latitudes there is however still quite a bit of focus on some areas and neglect of others. In particular the islands of the Kara Sea and the East Siberian Sea as well as Bear Island and Hopen south of Svalbard are significantly less often covered than other areas at the same latitude. This is of course in a way a matter of efficiency since recording a scene containing only a small island is less useful overall than recording one that fully or mostly contains land surfaces.

For comparison and completeness here are the same illustrations for the previous years:

year day night day pixel coverage
2014 LS8, LS7 LS8 LS8
2015 LS8, LS7 LS8 LS8
2016 LS8, LS7 LS8 LS8


Now to the main subject, the first year of Sentinel-2. Public availability of images started in November but quite a few images were already recorded earlier and made available over time so i will use the same end cutoff as for Landsat (mid October) and an open beginning and call it a full year. This results in a bit of a bias towards the northern hemisphere since summer images for it are included for both 2015 and 2016.

Technically doing this for Sentinel-2 is much harder than for Landsat for multiple reasons:

  • Since ESA does not provide a bulk metadata download option you have to scrape the API for this.
  • The API – in addition to being very unreliable recently is also not always returning all the matching packages for a query – at least not the search API, the OData API is different.
  • Not all packages in the ESA database have footprint data.
  • Packages extending over the 180 degree meridian have invalid footprints.
  • As mentioned in my initial Sentinel-2 review the scene footprints of the early data sets were mostly bogus. ESA fixed that in July but many of the earlier packages have not yet been replaced.
  • Some early packages from 2015 have incorrect relative orbit numbers.

All of this together means quite a lot of effort is required to generate a halfway accurate coverage analysis. The following has still quite a few limitations:

  • It only includes descending orbit images. Since Sentinel-2 does not have thermal infrared bands nighttime acquisitions are fairly rare anyway – so because of the complications i left them out.
  • Images with missing footprint data or wrong relative orbit numbers are excluded (a few hundred overall).
  • Images with old style bogus footprints were clipped to the recording swath limits but not in orbit direction. Therefore coverage is systematically overestimated, in particularly in areas with relatively fine grained acquisition patterns. This can well be seen in the smaller green areas in America and Asia which in reality are significantly smaller than indicated by the illustration here.

Still overall this should give a fairly precise image of the coverage patterns and recording priorities of Sentinel-2 during the first year of operations:

As it is already widely known Sentinel-2 operations do not aim for a globally uniform coverage. Focus is – at least for the last year – on Europe, Africa and Greenland. With a 10 day revisit interval Sentinel-2 could theoretically record 35 images per year for any single spot on the earth surface – although volume wise probably not for all the land areas at the same time. Understandably during the first year it did not reach this outside the overlaps anywhere although for the priority areas (Europe and Africa) it did match or even exceed the recording frequency of Landsat. Elsewhere however recordings were more sporadic and the smaller focus areas seem fairly arbitrary, probably meant to cater certain vested interests. This is particularly striking in the Antarctic where the interior is mostly uncovered but there is a smaller mostly featureless area on the East Antarctic plateau which has been repeatedly recorded for some reason.

Apart from the Antarctic and northmost Greenland completely uncovered areas are mostly smaller islands.

What the image does not tell is how well time slots are managed in the areas where images are not recorded at every opportunity, in particular with regards to cloud cover. With Sentinel-2 the area where this applies is much larger (i would estimate about two thirds of the land surfaces compared to about one third with Landsat). My gut feeling is that the USGS is doing better here – which would not be surprising considering the much longer experience. It might be interesting to look at the cloud cover estimates of the images in comparison although these are fairly unreliable and produced with different methods so in the end probably not really that meaningful. And as mentioned before the larger image size of Sentinel-2 also means cloud cover based scheduling is more difficult here.


So what is to be expected for the next year. For Landsat not much is likely going to change. For Sentinel-2 the current statement is

Sentinel-2A is acquiring Europe, Africa and Greenland at 10 days revisit, while the rest of the world land masses defined in the MRD are mapped with a 20 days revisit time.

As indicated previously such statements, especially regarding the MRD, have to be taken with a grain of salt. The above would mean a significant increase in the recording frequency overall as well as a more uniform global coverage – the past year as illustrated above showed in parts more than a 1:2 difference between Europe and Africa on the one side and America and Asia on the other.


October 20, 2016
by chris

Sentinel-3 data – better late than never

In February i reported the launch of the Sentinel-3A satellite and now the data finally starts to get available to the public.

According to the published schedule we now get the OLCI level 1 data, the SLSTR level 1 data is going to follow in November and higher level products are indicated for some time next year.

Color rendering of one of the first public Sentinel-3 OLCI data sets

Some readers who follow earth observation satellite deployments and operations might be astonished since Sentinel-3 images have been shown quite frequently by various parties for months already. This is because since May data has already been made available to so called expert users. None of the satellite image experts i know of is apparently part of this illustrious circle – indicating the expertise to qualify for this is different from what is usually understood as being an expert. This data was called ‘sample data’ but unlike normal sample data this was produced on a regular basis for the whole time – a somewhat creative use of the term ‘sample’.

This whole procedure is quite remarkable since the regulatory requirement of the Copernicus program is clearly that all data from the Sentinel satellites will be made available to everyone without access restrictions beyond basic registration. By declaring the satellite – or more precisely the data processing system since the satellite passed its in-orbit commissioning review in July – not yet operational this requirement is apparently circumvented.

You might call this paranoia on my part but the signs that a lot of influential people around the Copernicus program are largely uncomfortable with the whole open data aspect is quite visible in many things. For example look at the dates for the different Sentinel satellites launched to date – there is a clear trend visible:

  • Sentinel-1A: Launch 3 Apr 2014, public data access since 9 May 2014 (1 month)
  • Sentinel-2A: Launch 23 Jun 2015, access since end November 2015 (5 months)
  • Sentinel-1B: Launch 25 Apr 2016, access since 26 Sep 2016 (5 months)
  • Sentinel-3A: Launch 16 Feb 2016, partial access since 20 Oct 2016 (8 months)

Of course the first data released for Sentinel-1A was highly experimental and there were massive changes in the whole data distribution system afterwards. But that’s natural considering the lack of experience with public data distribution.

In defense of those in charge – the volume of data that needs to be made available for download is quite significant. For Sentinel-2 there are about 200-300 ‘scene’ packages per day (the old 300km packages, not the new single granule ones) which – with an assumed average size of 5GB amounts to about 1-1.5TB per day. For Sentinel-3 estimates are 28.5GB (OLCI) and 44.5GB (SLSTR) per orbit which amounts to more than 1TB per day for the basic level-1 data – not counting higher level products or the different near-real-time and long term versions. And the near-real-time product is supposed to be made available within three hours of recording. Given the difficulties that already showed regarding reliability of the data distribution with Sentinel-2 it is not really astonishing they are not too eager to open this fragile infrastructure to the public with even more data. But the ressources that go into the public data distribution component of the Copernicus program of course reflect the importance this has for those in charge – and it is not that the requirement to scale to this level was not already clear several years ago.

The situation might also be related to the fact that Sentinel-3 unlike Sentinel-1 and 2 is operated by EUMETSAT and not ESA (though ESA will distribute parts of the data). I wrote about the lack of an open data culture in ESA before but due to their scientific mission they at least superficially have to try to appear open.

EUMETSAT on the other hand is an intergovernmental organisation for running weather satellites for the European national weather services. Since it is inefficient for each of the countries in Europe to operate their own weather satellites they joined together. The concept of EUMETSAT is that countries joining it get free access to the satellite data for the national weather services for their own use. They are also allowed to license this data to third parties as licensing agents but are bound to the fees set by the EUMETSAT management. So essentially EUMETSAT has two purposes:

  • cost reduction by operating weather satellites together for the national services
  • monetarizing on the data produced by selling it to commercial users as a cartel

As a result policies for European weather satellite data are among the most restrictive world wide because it is explicit policy of the operator to try monetarizing on any uses of the data for non-governmental purposes – contrasting quite extremely for example to the Japanese with Himawari 8 which has full free public access to the data.

You might notice the irony in such an organization now being put in charge of operating a satellite with mandataory full open data access. Sentinel-3 does not compete directly with geostationary weather satellites of course but EUMETSAT is also operating other polar orbiting satellites.

Regarding the data itself – i have not looked at it in detail yet, will probably do a more thorough review later but likely not before also the SLSTR data is available. Since it comes in a non-standard form (netCDF format without normal georeferencing information) it will be fairly complicated to use with normal tools. What can be said so far is however that what is available is fairly incomplete. Since OLCI is visible and NIR wavelengths only night acquisitions are not of much use but they do not even have full coverage of the sunlit part, compare the footprints:

with the current Terra-MODIS coverage:

The more narrow swath and larger gaps near the equator are normal and expected but the fact that coverage ends fairly early towards the poles is not. If this is based on a sun elevation cutoff or some other criterion is unclear to me – just like if this is only a limitation to the data made available or if this represents the actual acquisitions. This will however likely limit data at high polar latitudes to a very small time frame of just 3-4 months per year.