Imagico.de

blog

sat_coverage_980

May 16, 2017
by chris
0 comments

Some satellite image news

It has been some time since i wrote about news in the world of open data satellite images and quite a few things have happened since then some of which i want to comment here.

Sentinel-2

Recently ESA published a report on the access to satellite data from the Copernicus program. This is mostly boring read with a fairly bad signal-to-noise ratio but there are also quite a few interesting things buried under a lot of meaningless all numbers have increased during the last year stuff. The most important thing to keep in mind if you read this is that the distribution form of the Sentinel-2 data changed to single granule packages during the reporting time frame. This is not properly taken into account so most numbers referring to package counts are nonsense combining apples and cherries so to speak. But to avoid misunderstandings – it is already quite positive that such a report is at least written and especially that it is published in the first place.

What is particularly funny is the following illustration meant to indicate the spatial distribution of published Sentinel-2 images.

For comparison here the accurate coverage map (which i published earlier for a different reporting timeframe).


The ESA illustration is not only combining the single and multiple granule packages apparently, it also senselessly blurs the illustration – probably they had the same troubles producing an accurate visualization from their own inconsistent metadata as me. Overall great example how not to do a visualization.

Some interesting facts that can be extracted from the report are:

  • a fairly open admission that availability of the open data access sucked beyond belief during the last part of the reporting time frame – something i already pointed out here. This is rationalized in the text of course. From current use of the service i predict that this will improve in the report for 2017 but the bar is not that high – they consider everything above 95% as good. And formal availability of the service as a whole of course does not necessarily means it is practically usable.
  • some interesting numbers regarding the actual use of the download services. I will get to these in the following.

Use of Sentinel-2 data access

I will only discuss Sentinel-2 numbers here while the report also covers the other Sentinel satellites of course. The report also covers both the Open Access Hub available to everyone and the other data hub instances available only to certain privileged groups:

  • the Copernicus Services Hub which is for organizations performing services inside the Copernicus program, i.e. stuff directly financed within the program.
  • the Collaborative Hub which is for partner organizations within the European Union, in other words independently tax financed stuff.
  • the International Hub which is for partner organizations outside the EU – which is currently one from Australia and two from the US (of either NASA, USGS and NOAA – which is not specified)

Now the main numbers: A total of 0.46PB of Sentinel-2 data has been published. Downloaded via the Open Access Hub by the general public were 1.53PB and via the other non-public access hubs were 1.14PB.

Of the downloads by the general public about 75 percent were data that was less than one week old, there were about 6500 registered users accessing Sentinel-2 data of which less than 100 downloaded more than 100 packages.

Now my interpretation of these numbers:

  • The routine very large volume data users (think: Google, Amazon – but also various smaller ones probably) apparently do not get their data via these channels, they must have separate arrangements which are not publicly reported on.
  • Independent data users with a larger volume of data use like myself are extremely rare. Almost all of the use via Open Access Hub is just testing a few recent images and no routine use. Of course this is the first year of operations and people just start getting interested. And naturally the frequent changes in data format and the bad reliability of the systems does not really encourage use and especially smaller data users will likely wait if this stabilizes and use alternative data sources in the meantime.
  • If there is significant large volume access to the Sentinel-2 data from the partner organizations through these channels it must have started relatively late in 2016 since overall numbers likewise suggest mostly limited volume use.

Some additional data

Here some additional illustrations based on the publicly available metadata which are not in the report and which are also not widely discussed otherwise. First the development of published image volume in terms of covered area with Landsat in comparison.

average daily acquisition volumes of open data satellites

Note this is not a hundred percent accurate due to the difficulties of accurately calculating it from the available metadata. But it is pretty close. The volume is always calculated per orbital period, the repeat cycle of the orbit pattern which is 10 days for Sentinel-2 and 16 days for Landsat. The numbers are then normalized to a daily volume for comparability.

What you can see is that Landsat 8 has been recording on a relatively stable level since 2014 between about 20 and 23 million square kilometers per day. There is a seasonal pattern due to the variation in illumination of the earth land masses. Landsat 7 follows a different pattern on a lower overall level with a minimum in the northern hemisphere winter since it does not record the Antarctic. If you look closely you can also see a drop in last winter in the numbers of Landsat 8 which comes from the Antarctic coverage being limited in the 2016/2017 season mostly to coastal areas for some reason. I could not find any documentation of this in the USGS materials and I hope this is a one time incident and will not mark a permanent change in the recording patterns.

Sentinel-2 numbers are meanwhile on the same level as Landsat 8 in terms of overall coverage volume but the numbers are much less stable with many irregularities and gaps. This brings me to the next thing i prepared – which is a visualization of the coverage by orbital period indicating the images available as well as those missing – images which were planned to be recorded according to the published plans but which are not available for download – either because they have not been acquired or because they are not processed.

It should be noted that the way acquisitions are planned differs strongly between Landsat 8 and Sentinel-2. Landsat 8 has a dynamic acquisition system based on predefined scene priorities that leads to an automatic short term selection which scenes are recorded based on a large number of influencing factors and the resulting acquisition plan is usually very close to what is actually recorded and then also available. Sentinel-2 in contrast to that is operated with a fixed acquisition plan set long in advance.

In any case what can be concluded from this is that there are significant derivations of actual operations of Sentinel-2 from the published plans and the recording numbers (in minutes per orbit) listed in the mission status reports are not actual recording numbers but just what has been planned. I am not sure if those gaps which are clearly missing recordings are due to the acquisition plans overbooking the satellite and data transfer systems or if there are outages in components of the data transfer system or operational errors. The missing recordings are too frequent to be purely due to operational contraints like orbital maneuvers, calibrations etc.

In addition gaps in processing with individual granules missing are fairly common and these do not get filled after a few days as you might expect. If these would be filled by later reprocessings of the images is an open question. This seems to be a side effect of the move to single granule packages – i never experienced individual granules missing with the larger packages – paired with a lack of fault tolerance and error checks in the processing system.

Last i also fixed the daily scene numbers page which had not been working correctly for quite some time. These numbers are now also normalized area coverage numbers.

With the whole thing it is of course important to keep in mind that the spatial resolution of Sentinel-2 imagery is higher than that of Landsat so the data volume for the same coverage area is naturally significantly larger.

Landsat

I kind of fear that with ESA getting all screen time here with critical commentary the USGS might get jealous so here also a few words on Landsat (although i already had a bit of analysis in the previous paragraphs).

Transit to the new ”Collection” distribution form for Landsat data is now complete. As mentioned previously this is a fairly superficial change for most data users. The USGS also continues to use the old scene IDs in their data management system at several places – for example the metadata pages still have the form

https://earthexplorer.usgs.gov/metadata/12864/LC81920272017130LGN00/

The most significant shortcoming from my perspective is however that the bulk metadata they make avilable is apparently incomplete – at least for Landsat 8 there seem to be more than 3000 scenes missing in the Collection data set which are in the old legacy database. This appears to be a problem of the metadata file though – the scenes appear to be available in EarthExplorer.

amery_osm_980

May 1, 2017
by chris
0 comments

OSM map update cycles

Recently i made some effort updating the Antarctic coastline in OpenStreetMap regarding the most significant changes due to the movement of the ice. This essentially means bringing the position of the outer edges of the three largest ice shelf plates to the current state. These ice edges move by about 1-2 km per year, i had last updated them about two years ago and in between no one else has significantly modified the data. There is not that much additional value in mapping these in fine detail even if suitable images are available so i used fairly low resolution images as basis. Other parts of the Antarctic coastline have not been updated in recent years so are fairly outdated but probably more than 80 percent of the area change happens in these sections.

Amery Ice Shelf coastline change in OpenStreetMap

For illustration above the history of the coastline at the Amery Ice Shelf edge in OpenStreetMap. This kind of regular change with updates from time to time in the database makes these areas a great test case of evaluating the update cycle of maps. The shown changes happened early 2013, mid 2013, early 2015 and the last one about two weeks ago. Most OSM based maps show changes in the base data pretty fast, sometimes after minutes. But this only applies to normal small geometries and only to the higher zoom levels. Low zoom updates tend to be less frequent and coastline updates likewise. The map on osm.org usually updates the coastline data on a daily basis but the low zoom levels up to z12 are only rendered once per month. Many other map services are even slower. Here a Map Compare view of various maps of the area discussed.


Right now the German OSM style on openstreetmap.de seems to be the only widely used one that is up-to-date with the coastline at the low zoom levels two weeks after the edits have been made. The map on openstreetmap.org is currently behind in updates due to hardware problems but would normally also show the current data at least on the higher zoom levels. As you can see many of the other map styles do not even show the 2015 update although on the positive side it should be noted that no map style i checked with some importance is using a pre-ADD (before mid-2013) data basis. In the coming weeks and months you will be able to observe if and when the various map providers update their coastline data basis.

If you render maps based on OpenStreetMap data you can find recent coastline data on openstreetmapdata.com.

patagonia_autumn_crop1.ann

April 22, 2017
by chris
0 comments

Autumn colors in the south

After the recent work on autumn color satellite image mosaics here for comparison also an image of autumn colors on the southern hemisphere.

This shows the Andes Mountains in Patagonia, South America at the border between Chile and Argentina with the Northern Patagonian Ice Field in the lower part of the image on April 16 this year. In general autumn color changes of the vegetation are much less widespread on the southern hemisphere than in the north. In this case the most widespread color change occurs with deciduous Nothofagus forests which grow at higher altitudes just below the tree line in the drier, eastern parts of the mountains. These trees feature a color change to strong red in autumn which can be prominently seen in this image. The western sides of the mountains and the fjords at the coast however are dominated by evergreen trees and do not change color that much. Here a few detail crops.

You can compare the last one also to the second image i showed with a talk announcement recently which was taken almost exactly a month earlier.

This image based on Copernicus Sentinel-2 data can be found in the catalog on services.imagico.de.

osmim11_980

April 18, 2017
by chris
0 comments

Images for mapping update 11

I just uploaded another update for the OSM images for mapping containing the following new imagery:

First there is a new low tide image of the North Sea Coast of Germany, Denmark and the Netherlands from September last year suited for updating the tidal flats, beaches and shoals in the area. These change quite rapidly so even if you have full coverage in high resolution images not extremely old it is a good idea to consult recent images when you map there to avoid investing in mapping features that might no more exist or look very different now.

Then there is an image of the Pechora Sea Coast in northern Russia – likewise from last September. This allows improving the mapping of the coast and tidal areas and adding missing lakes but also shows quite a few roads not currently mapped in OSM.

I also decided to add recent coverage of a small part of the Antarctic coast, mostly to demonstrate that the LIMA images which are shown by all other image sources there are not really useful to improve of the existing data in the area because they are hopelessly outdated where ice is moving. But even if the images here are recent and from late summer with minimal seasonal ice you should familiarize yourself with the area before remotely mapping based on images alone – it is not always easy to distinguish between permanent and non-permanent ice for example. In case of doubt just concentrate on obvious features, in particular updating the glacier front positions and correcting and detailing ice free rock outcrops.

And finally i also added the recently introduced Svalbard mosaic as an up-to-date and accurate basis of mapping Svalbard. So far glacier mapping on the islands is essentially missing in OpenStreetMap and large parts of the coastline are still extremely crude. This image will give anyone interested plenty of up-to-date information for accurate mapping of the area.

sib_autumn_980

April 10, 2017
by chris
0 comments

Taming the Chameleon

To illustrate the difficulties of assembling cloud free satellite image mosaics i tend to use the analogy of trying to photograph an elephant running through a dense forest at a distance and changing its color like a chameleon while doing that.

This metaphor is meant to illustrate the two primary difficulties of image assembly – the dense forest stands for the clouds while the chameleon nature illustrates the changing appearance of the earth surface over time. Now most efforts in satellite image assembly and most research and discussion going into that tends to focus on the clouds – they are an universal and unavoidable problem you have to deal with when working in this field. The changing appearance of the earth surface on the other hand is not usually given much consideration. A quality satellite image mosaic needs to deal with that as well but usually this tends to be treated as a side effect within the bigger cloud related problems and people just try to handle this but don’t care much about how they handle this.

There are a number of strategies that are commonly used when specifically dealing with this matter. Most of them tend to select a certain state within the spectrum of appearances and target what i like to call a stable extreme in the earth surface appearance. Targeting the vegetation maximum is an obvious and widely used approach here. Stable extreme means that that (a) the appearance you target is an extreme state with regards to some kind of definable property and (b) that this state is stable to some extent, i.e. nature has some persistence in staying this way. The vegetation maximum in most parts of the earth qualifies as such a state. Another frequently targeted stable extreme, especially in regions with a distinct rain season, is the dry season because clouds are often less of a problem then. A less stable extreme would – at least in temperate climate – be the snow maximum. Snow cover in low land areas in central Europe tends to be fairly volatile.

The most difficult state to target are the unstable transient states. One of the most interesting examples for this are autumn colors. The difficulty of assembling image mosaics showing this kind of situation is that

  • you essentially target a single point in a fairly rapid development.
  • what you target is not easily quantifiable because the change in colors of the vegetation during autumn is a multidimensional effect, different species of plant change their colors at different times across different color ranges and depend on the local setting in this regard as well.
  • how this happens and when exactly this happens often varies strongly from one year to the other and the variance in time is often not small compared to the duration of the whole process. In other words: when the leaves start to change their color in one year they might already be falling in another year.

Because of these difficulties image mosaics of larger areas showing autumn colors are very rare. Trying to find out what can be done in that field i produced the following image of northeastern Russia showing the lower Lena River and the Verkhoyansk Range with autumn colors.

This by the way is the region on Earth with the most extreme temperature differences between winter and summer with a total range of naturally occurring temperatures of more than 100 degrees Celsius in some parts. It also is a region with extensive and quite diverse color changes of the vegetation in autumn – which is why i choose this area here.

Compared to the usual image mosaics i produce creating such an image is not a well established and routine process yet. As you can see i did not create full coverage of the rectangular area and consistency as well as the level of cloud free-ness of the image are not quite on my usual levels. Due to the difficulties i described above many of the techniques i normally use for image assembly that help ensure the quality level do not work that well when i target an unstable transient state. None the less i think the results are pretty decent considering the difficulties. Here a few magnified crops:

Data basis is mostly Copernicus Sentinel-2 imagery supplemented with about 20-30 percent Landsat data.

ant2017_980

April 8, 2017
by chris
0 comments

Spring again

Spring is coming again around here while winter is approaching on the southern hemsiphere. And as last year a few images from the Antarctic before the night and from the Arctic from the beginning of spring:

The first one shows the western part of James Ross Island on the east side of the Antarctic Peninsula with the ice free Prince Gustav Channel between the island and the mainland.

The second image features Mount Takahe in West Antarctica, one of the largest of the extinct volcanoes of Marie Byrd Land peeking through the clouds in the last rays of the autumn sun.

And then from the northern Hemisphere i have a kind of Yin and Yang image of northwestern Svalbard

In this area ocean currents are transporting warmer water to the north so the area at the western coast is mostly ice free. Winds from the north lead to clouds forming over the warm water but air temperatures are fairly high already so there is not much formation of new sea ice. The north of Svalbard however is still tightly enclosed in ice.

And finally here an image from the southern Kamchatka peninsula of the eruption of Kambalny:

For comparison see also my former images of Kamchatka volcanoes.

First and third image are made from Copernicus Sentinel-2 data, second and fourth from Landsat 8.

fossgis_2017_980

March 20, 2017
by chris
2 Comments

FOSSGIS 2017 and talk announcement

On Wednesday this week the annual FOSSGIS conference is going to start in Passau and I look forward to be there and meet people and talk about free software and geodata. On Friday morning i am going to present a talk on free satellite images giving an introduction on what is available today in terms of open data satellite imagery for everyone to use.

As a teaser here a few images i am going to show as examples. These can be used under CC-BY-SA license, you can also print them in large if you want to. Be careful – these are fairly large image files with more than 6000 pixel in size.

Verkhoyansk Range, eastern Russia 07-09-2016 by Landsat 8 – full size image

Patagonia, Chile 15-03-2017 by Landsat 8 – full size image

Mt. Katmai/Novarupta, Alaska 13-09-2016 by Sentinel-2 – full size image

As a bit of a side blow here some links to what the usual suspects offer in these areas – Patagonia, Siberia, Alaska

Those who want to come by spontaneously can do so – Entry is free for those active in Open Source software and free geodata.

Update: Video of the talk is now available on youtube, a PDF of the slides will also be linked from the program later.

icebridge_southpole

March 17, 2017
by chris
0 comments

Lost and Found

After cleaning up remaining broken multipolygons in OpenStreetMap in the Antarctic as part of an ongoing effort to fix broken geometries – by the way I am happy to report Antarctica is now the first continent in OpenStreetMap without serious multipolygon errors or old style tagging of multipolygons – i also did a traditional cleanup round around the poles to remove bogus data.

The phenomenon of accumulating garbage in the OSM database is best known from Null Island. But sometimes the data also turns up around the poles and accumulates there because people rarely ever look there and clean up. Areas beyond the Mercator map limit slightly beyond 85 degrees latitude do not turn up in most QA tools and editors so they are kind of invisible to normal OSM activities. I have not done such a cleanup for some time and apparently others have not either so here are a few highlights of things that got lost there.

Interestingly if you look at the changesets which created those – most of them were created with iD – although to my knowledge iD works exclusively in Mercator projection. So you apparently can generate data near the poles with iD but you have no way to edit it after you have done so. Here a few changesets as examples:

33412666, 39331559, 42690278, 41522831, 39111625, 16380595, 37911018

Now the area around the south pole is pretty empty again. It is difficult to map here because although JOSM can meanwhile be set to use polar projection (EPSG:3031 for the Antarctic if you want to try) none of the usual image sources covers this area. If you want to do some mapping around the south pole here is an Icebridge image from a few months back. You can download and use it in JOSM using the ImportImage plugin after setting the projection to EPSG:3031. What does not work in this case is getting the existing OSM data via API – you need to use Overpass/XAPI for that.

To identify the different things you can see on the image – here is a plan of the area and an annotated oblique image.

Note the Icebridge image is from very early in the summer season so there is relatively little visible except the permanent structures and traces of past activities have been largely covered by snow and wind during winter.

Another thing to keep in mind: everything there is located on ice that moves by several meters every year so it is important when you map things to specify the date of the location information – in case of this image October 2016.

EO1A1050702017052110KF_expose.ann

March 4, 2017
by chris
0 comments

EO-1 unending

When i wrote my eulogy on EO-1 i closed mentioning that final decommissioning is scheduled for late February but after having defied odds for more than 15 years it is only fitting for this satellite that this was not the final word on the matter. The new date is now announced to be around March 20 – lets see how this is going to work out. In the meantime enjoy more spectacular early morning views.

S2A_R024_S56_20170213T140721_expose.ann

March 3, 2017
by chris
0 comments

The truth about true color images

I have been meaning to write a piece about this for some time already and a post on the OpenStreetMap user diaries reminded me about that. Recent changes in the distribution form of Copernicus Sentinel-2 images – which I wrote about in relation to other aspects previously – also introduced something I did not write about yet – the full resolution True-Colour Image. I did not discuss this because it was not of much concern for me. I disliked the additional download volume but otherwise this would not cause any problems. Later I however realized that for many beginners in using satellite imagery this will probably have a much higher impact because they will often be inclined to actually use it and might even view the data exclusively through this image.

The True-Colour Image is essentially a large, full resolution version of the preview images you can find in the ESA download application and can also query through the API. I made critical remarks about the rendering of these previews before and those essentially apply to the full resolution version the same way. The developer who planned and implemented generation of these images quite clearly did not know much about either satellite imagery or color representation in computers in general on the current technical level.

What today’s satellites – including Sentinel-2 – produce as raw imagery is pretty high quality data, not only in terms of spatial resolution but especially also in terms of dynamic range and low noise levels. Even if you just look at the true color channels, i.e. red, green and blue, this data cannot be fully reproduced on a computer screen, you need to compress the dynamic range available into the range supported by computer displays. Doing this is not a simple task, it requires knowledge of color representation, image processing and color physiology and ideally it takes into account what you want to use the image for. Still the way this is done with the Sentinal-2 True-Colour Images is about the worst possible way this can be done. Not only does this immensely reduce the usefulness of these images for the user, it also significantly sells short the quality of the underlying data.

Here an example from Patagonia near the southern tip of South America:

For comparison a custom rendering produced by me from the raw data:

Clearly visible are the clipped highlights in the first image which makes it hard to distinguish clouds from snow and the fairly structure-less shadows where you can hardly see anything. Both of these are not problems of the data but of the processing applied – as evident in the custom rendering.

Now you might say that it is obvious when you compare a static processing with one specifically adjusted for the setting but this is not the point here – you can do much better even with a globally uniform rendering applied to all images identically. And using the poor rendering of the color composite images in the Sentinel-2 packages you loose a lot of valuable information that is actually in the data. Or to look at it from a different perspective – apart from the higher spatial resolution you could produce this kind of rendering also from a 1980s Landsat 5 image.

The color fringes around the clouds are not a processing artefact by the way but due to the way the satellite records images.

s2sval_980

February 9, 2017
by chris
0 comments

North Atlantic island images

Over the winter I processed various new satellite image mosaics making use of new data from the 2016 summer and I am pleased to introduce some of these here. As usual you can find more detailed information on services.imagico.de and you can contact me in case you are interested in using one or more of these images for your own applications.

Svalbard

I already introduced a Svalbard image in 2015 which was and still is the highest quality image of this kind available regarding uniformity, color consistency and lack of clouds. But there always is room for improvement of course. Here is a new image of the same area based on Sentinel-2 data from 2016. The most obvious improvement is the higher resolution of course but this is not the only difference.

Sentinel-2 mosaic of Svalbard

The new image is nearly all from Sentinel-2 data from just one summer season which is somewhat astonishing considering the previous image used three years of Landsat 8 data and still required some Landsat 7 images in addition. The reason for this lies in the somewhat peculiar operation concepts of Sentinel-2. While Landsat operations try to get a nearly uniform 16 day coverage of all land areas and therefore often skips recording opportunities at high latitudes where they are not required for the 16 day interval, Sentinel-2 did not use such a rule last year in Europe leading to a very high recording frequency over Svalbard during summer. See also the coverage maps I showed some time ago. This produced a lot of fairly worthless images since there are lots of clouds in Svalbard, especially in the summer. But it also produced a higher number of images and more complete coverage during the few good weather windows in late summer last year.

This is something I have mixed feelings about. Of course it is nice for the Svalbard area but if you consider how this recording capacity could be used otherwise in areas that are currently only recorded with low priority, in particular Asia and South America, this is ultimately a fairly questionable strategy on a global level. But this is of course a political decision at ESA and there is very little chance that those making it are receptive for global and long term considerations – people do not get in such a position by putting these things first.

Also included in the Svalbard mosaic as a separate image is Bear Island which i did not have in the 2015 image because there was not enough good quality data here at that time.

Sentinel-2 mosaic of Svalbard – Bear Island

Iceland

The other large mosaic is of Iceland. Iceland is among the hardest areas on the northern hemisphere outside the tropics in terms of clouds in satellite images. When I produced 3d views of Iceland previously I heavily relied on Landsat 5 images which I did not use in this mosaic due to their age and low resolution. There is still quite a bit of room for improvements, especially regarding a tighter late summer time frame with a minimum in seasonal snow but it is the first time it was possible to produce an image of this Island within my quality standards.

Landsat Mosaic of Iceland

Jan Mayen and the Faroe Islands

And then there are two more small mosaics of the other Islands in the North Atlantic, Jan Mayen and the Faroe Islands. Jan Mayen is based mostly on Sentinel-2 data while the Faroe Islands are mostly produced from Landsat images.

Sentinel-2 mosaic of Jan Mayen

Landsat mosaic of the Faroe Islands

You can click on the images above to go to the detailed description on services.imagico.de.

EO1_erebus_980

January 20, 2017
by chris
0 comments

Antarctic summer midnight sunrise

As a followup to my recent EO-1 post here another set of unusual images from this satellite from a few months ago.

These kind of show the start of the polar day in the 2016/2017 summer in the Antarctic. At these latitudes (about 77.5 degrees south) most of the year is either dark all day (polar night) or with permanent light (polar day). These images are from the short timespan in between when you actually have a sunrise and sunset. To be precise these actually all depict sunset with respect to the daily move of the sun but as a sequence they illustrate the end of the transit from polar night to polar day.

All these images show Mount Erebus on Ross Island with the Hut Point Peninsula where research stations from New Zealand and the United States are located.

2016-10-20

2016-10-24

2016-10-28

2016-11-01