Imagico.de

blog

October 10, 2021
by chris
0 comments

Autumn and Spring in October 2021

Together with the two Landsat image visualizations i showed for the Landsat 9 launch featuring northern hemisphere autumn impressions i have added two more Sentinel-2 images to the catalog on services.imagico.de as well.

The first is a fairly clear early spring image of South Georgia showing the beginning of the thawing season.

South Georgia in early October 2021

South Georgia in early October 2021

The second is a nice autumn view of Mount Aniakchak on the Alaska Peninsula in late September with some fresh snow around the rim of the caldera contrasting with the darker surrounding.

Aniakchak in late September 2021

Aniakchak in late September 2021

September 27, 2021
by chris
0 comments

Landsat 9 launch

Today Landsat 9 was launched. As i mentioned before i typically do not discuss Earth observation satellite launches here because the more interesting and practically meaningful date is when the data starts becoming available. But i make an exception for Landsat because of its historic and present day significance for Open Data satellite imagery.

Landsat 8 image recordings for 2020 – see here for more details

Since images from Sentinel-2 started to become routinely available Landsat has kind of lost significantly in attraction to a lot of data users primarily because of the higher spatial resolution and higher revisit frequency of Sentinel-2. However as i have pointed out on various occasions of the two systems Landsat is the only one with a more or less unbiased coverage of all land surfaces of Earth while Sentinel-2 recordings are scheduled based on short term political and economic interests. And with a second satellite providing high quality data and thereby reducing the uniform quality revisit interval from 16 to eight days Landsat certainly is going to get more attractive.

Still of course with Landsat 9 being more or less a copy of Landsat 8 – which is 8 years old already, probably does not seem like the most exciting news to many. What comes after Landsat 9 is still unclear at the moment – there seem to be some studies on possible concepts running at the moment and some not very specific considerations have been published. For quite some time there were indications that Landsat 9 might be the last Landsat producing completely open data and that the US government would look into privatizing the program. With the current US administration giving larger budgets to publicly financed earth observation that seems less likely now but what direction this will go in is unsure as ever. In light of that the boring nature of Landsat 9 offering merely continuity in available data is not necessarily bad news.

Here – to give an idea what to expect from Landsat 9 – two examples from the past few days from Landsat 8.

Mount Katmai and Naknek Lake, Alaska – September 26, 2021

Landsat 8 - Mount Katmai and Naknek Lake, Alaska

Landsat 8 detail

Landsat 8 detail

Northern Greenland – September 13, 2021

Landsat 8 - Northern Greenland

Landsat 8 detail

September 25, 2021
by chris
0 comments

About the lack of progress in quality of satellite image visualizations

Satellite images and their use have become much more popular in recent years. There was already an earlier wave of broader popularization of satellite imagery with image layers being widely introduced in popular map services on the internet between about 2005 and 2010. This for the first time exposed a large number of people to satellite imagery as a source of information and spatial orientation and not just as mere anecdotal photographs.

What i am talking about here however is the more recent trend of the past roughly five years – so about ten years after the first wave. In contrast to the first wave which was primarily about casual use of imagery this second wave of popularization of satellite imagery involves in particular also more serious or semi-serious active use of satellite data by both professionals and hobbyists with diverse (and often not image processing and not earth observation related) backgrounds. Driving this recent trend is in particular the advertisement and PR which satellite operations – both private businesses and public institutions – invest to create interest in the use of their data and services. This combines with many providers of cloud based services trying to attract paying customers for use cases involving satellite data.

Some might disagree with me identifying these two distinct waves of popularization and instead would see a continuous trend. But i see a distinct period of stagnation in popularization between these two phases and i have been actively observing the field over the whole period so i think i have a pretty good read of it.

Anyway – what i want to write about here is not so much these waves of popularization of satellite imagery but how quality of satellite image based visualizations has developed during this. The technical quality of satellites’ image sensors has improved massively over the years – i have written about many examples showing this. Interestingly this trend in satellites is paralleled by a similar (and technologically related) trend in ground based digital photographic technology as well as its use. To illustrate the point i want to make about satellite image visualization i will therefore make a quick excursion into the development of digital photography over about the same time period.

Since the beginning of mainstream digital photography in the late 1990s sensor and camera technology have seen a similar quality development as satellite image sensors. And the use of the image data produced by these cameras has seen a similar development. With that i am not talking about exotic methods employed by a small elite of experts, i am talking about mainstream image processing methods available to and used by many photographers. Based on this development both in sensor and camera technology as well as in processing methodology even a cheap mobile phone camera will produce images out of the box that would have made a professional photographer of the early 2000s using the most state-of-the-art equipment envious. And with some rudimentary learning and training in use of broadly available tools and techniques (either in-camera or in post processing) you can easily improve in the quality of ground level photographic data visualization if you want so far beyond that.

example of the technical level of the quality of ~2005 consumer digital camera technology with low dynamic range, high noise levels and poor color depiction

Example of the technical level of the quality of ~2005 consumer digital camera technology with low dynamic range, high noise levels and poor color depiction

out-of-the-box results from a present day (2020 model) mobile phone builtin camera showing excellent dynamic range and low noise, very decent color rendition (which can be much improved on with larger image sensors available these days) and an equally decent automated visualization for an sRGB color screen

Out-of-the-box results from a present day (2020 model) mobile phone builtin camera showing excellent dynamic range and low noise, very decent color rendition (which can be much improved on with larger image sensors available these days) and an equally decent automated visualization for an sRGB color screen

Getting back to satellite images now – as said the technical development in satellites’ camera technology pretty much reflects the sensor and camera technology development in ground level cameras. But in data processing and visualization technology employed in broad popular use it does not. Since at least the start of Landsat 8 in 2013 – which marked a significant step up in quality of open data imagery available in larger volume – i have been amazed by the stagnation and sometimes even decline of technology and sophistication in processing methods in the visualization of satellite image data. I have written about this previously in a specific case which – through non-sophisticated processing in visualization – excessively undervalued the quality of the image data used. But this is just a single case example of a much broader phenomenon.

Mediocre quality visualization with clipped colors underselling the much better underlying image data

Mediocre quality visualization with clipped colors underselling the much better underlying image data

Better visualization of the same data

Better visualization of the same data

The amazing thing is that this is evidently not a problem of missing innovation because the innovations not used that would be of value to produce much better visualizations have already been developed and are broadly used – in the domain of ground level photography.

A large fraction of the many people working with satellite image data these days and proudly presenting visualizations of such as testimony to their expertise in the domain work without making use of some of the most basic technological and methodological innovations that have been developed for digital image data visualization over the past decade in photography and that are now available routinely in every phone.

Poor visualization of recent Etna activities from Sentinel-3 OLCI data with distorted colors and unnatural color clipping

Poor visualization of recent Etna activities from Sentinel-3 OLCI data with distorted colors and unnatural color clipping

More consistent visualization of the dame data giving a more balanced and more accurate impression of the setting

More consistent visualization of the dame data giving a more balanced and more accurate impression of the setting

So in a way you could for many of the mediocre visualizations you can see these days both on social media and more serious channels essentially comment: The 1990s called, they want their crappy, distorted colors and overexposed JPEGs back.

As for the reasons for the described phenomenon – a huge part of that is probably because people predominantly do not view satellite images as photos but as a more abstract type of depiction.

The reason why even in the most low end sectors of the digital photography market we have over the past two decades essentially had a continuous competitive pressure for improvements in technical quality is because users of these devices and related software assess and judge quality and are able to do so through comparison with their direct personal visual experience. That is not the case with satellite images. To put it very simply: People tend to expect upfront that satellite image visualizations in many ways do not match or are consistent with their personal viewing experience of the areas shown on the images. Therefore they are more inclined to accept badly made visualizations of satellite images.

That does not mean people are unable to appreciate quality in satellite image data visualizations if they are confronted with the difference like in the examples above. So in a way by aiming for excellence in visualization of satellite imagery you show respect for the needs of your recipients without requiring them to articulate those needs as demands.

September 13, 2021
by chris
0 comments

Satellite image news

The title of this section should be interpreted loosely this time – i have not discussed new developments in the domain in general for some time now so some of the news are more than a year old already.

There were a number of of recent and are a number of upcoming satellite launches both in the Open Data domain (in particular Landsat 9 of course) and in commercial Earth observation. In the latter we have had in particular the first two of four Pleiades Neo satellites launched (May 20 and August 16) which are going to offer increased resolution and higher recording capacity compared to the existing two Pleiades satellites. Maxar has been showing their plans for a new constellation of more lightweight high resolution satellites making use of economics of scale in building them and offering more flexibility in recording than single and more heavy system like they are predominantly using right now. But there seems no definitive launch date yet. Still the Pleiades Neo launches evidently will put pressure on them. Planet Labs has also started launching an updated version of their micro-satellites offering additional spectral bands. As usual they have been very tight-lipped in public about this though.

In terms of newly available open data satellite imagery i want to mention in particular the Brazilian CBERS-4A (launched December 20, 2019) and Amazonia-1 (launched February 28, 2021) satellites. It seems the data released by the National Institute for Space Research (INPE) is all available under CC-BY-SA which is more restrictive than the licenses of other open data image providers but still the most notable entry in the open data imagery domain from outside Europe, North America and Japan so far. What is not clear is if the data published by the INPE is all the data recorded by the satellites or if they release only data selected to be published as open data and there is also unreleased data that is not available freely. Considering the data published is mostly limited to South America that is not an unlikely scenario.

Amazonia-1 image example over southern Peru/Bolivia - whole scene

Amazonia-1 image example over southern Peru/Bolivia – whole scene

The Amazonia-1 satellite with its 850km swath and 60m resolution fills a noticeable gap in the European, North American and Japanese systems between the high resolution systems (resolution better than about 50m but swath width less than about 300km) and the low/moderate resolution systems with swath width of more than 1000km but resolutions of 250m or worse.

Amazonia-1 image sample - magnified crop

Amazonia-1 image sample – magnified crop

The actual Amazonia-1 image sensor is not particularly advanced by today’s standards – dynamic range seems to be comparable to Landsat 7 – with the same tendency to saturate the sensor on bright features. It seems to be using 10-bit AD conversion.

Amazonia-1 image sample - full resolution crop

Amazonia-1 image sample – full resolution crop

There also seems to no radiometric calibration data available so far – just the raw sensor values. But still this is a fairly impressive source of imagery available there with the combination of a fairly wide field of view with a moderately high resolution and it would be great to have this available also for areas outside the region in South America covered so far.

Targeting the same gap in available imaging system is another experimental system – SeaHawk – that was launched in 2018 already and that became more prominently publicized recently. SeaHawk is a less than 5kg micro-satellite that records a 216km swath at 120m resolution. That is less ambitious than Amazonia-1 and provides less nominal coverage than a single Sentinel-2 satellite but at that form factor this is pretty impressive, especially also in terms of image quality.

SeaHawk image sample from Eastern Greenland

SeaHawk image sample from Eastern Greenland

SeaHawk image sample from New Caledonia

SeaHawk image sample from New Caledonia

If you look at the images available more closely you can see various issues and flaws, in particular:

  • a significant level of banding and striping due to non-uniformity in the sensor. This could probably be partly compensated by better per-pixel calibration but still this is on a level significantly worse than larger current satellite sensors.
  • a significant decline in resolution towards the swath edges which is clearly the result of limitation of the optics and not of the view geometry (larger viewing distance at the edges – well known for wide angle systems like MODIS and VIIRS) alone.
  • a significant mismatch between the spectral bands leading to color rims at sharp contrast edges.
SeaHawk image sample magnified crop

SeaHawk image sample magnified crop

The most significant operational issue however seems to be the poor out-of-the-box geolocation accuracy of the images. The raw Level-1 data you can download comes without any geolocation data and so far they seem to be manually geocoding the data on the coastline. Micro-satellites like this have only very limited ability to determine their own orientation in space and therefore limited ability to know where exactly they are looking. That lack of accurate information severely limits the ability to properly and accurately geocode the image data. A more detailed discussion of the matter is provided by the operators.

In a quite similar direction of miniaturization goes the PROBA-V plus one project – meant to be a follow-up on the ended PROBA-V – which unfortunately despite being paid with tax money has never been fully open data. It will be a good test case to see if the organizational culture at ESA is changing in light of the Copernicus program (where the open data policy has been imposed from the top by the EU Commission) or if the influential people in and around ESA still prefer to keep the bulk of the data exclusive to themselves and those deemed worthy.

Following up on my posts on Sentinel-3 image use and the severe issues with geolocation data in the SLSTR files – ESA has in late 2019 apparently finally noticed and fixed the problem. Newer files produced (produced after early 2020, indicated by a ‘004’ version string in the file name) do not contain this error any more. Older Sentinel-3B data has also been reprocessed but Sentinel-3A data not yet.

I will follow up on this in a bit more detail in a separate post probably.

Of all the optical earth observations systems in the EU Copernicus program i had not yet looked at the Sentinel-5P satellite. It is of relatively little value for the work i am doing but for completeness reasons i had a closer look. Sentinel-5P features a low spatial resolution hyperspectral sensor which records data in particular about the atmosphere and focuses on the UV and blue domain. It would be great for considering and demonstrating the effect of different spectral band designs of multispectral sensors on visualization of the Earth but unfortunately its spectral coverage has a huge gap right in the middle of the visual range where you would typically have the green band.

Sentinel-5P L1B data false color visualization showing data from Sept 10, 2021 at around 305nm, 390nm and 475nm

Sentinel-5P L1B data false color visualization showing data from Sept 10, 2021 at around 305nm, 390nm and 475nm

The illustration above shows the visualization of the data of a single day in the UV-Range and in the blue spectrum in a false color display. The strips along the flight direction of the satellite are caused by the different scattering characteristics depending on illumination and view direction. A large part of the signal in the ultraviolet is caused by scattering in the atmosphere. Different gases and aerosols have different scattering and absorption characteristics depending on the wavelength – visible on that day in particular in the mid west of the USA where the smoke from fires shows up in strong red-brown tones. The Sentinel-5P sensor offers very detailed spectra of every pixel recorded. Overall more than 150GB of L1B data were processed for the production of this illustration (and that is just three of eight spectral bands).

Sentinel-5P by the way is the only satellite from the Copernicus program that covers the whole planet surface during a single day.

Finally a bit of seriously bad news – NASA has last year announced that the Terra satellite is nearing the end of its life and will – due to the lack of fuel (i discussed this in more detail for EO-1 in the past) – start to drift in its orbit. It has meanwhile exceeded nominal specifications for the recordings and will probably reach a 15 minute difference in timing/position of its images by September 2022 and will likely cease producing usable images by 2025 latest.

I will probably write a bit more about the Terra satellite and its significance for the historic development of Earth observation from space as well as its present day significance. In particular i would also like to discuss the MISR instrument. Most people know Terra in particular for the MODIS and ASTER sensors and MISR – being somewhat in the shadow of the more popular MODIS is one of the most underrated and underused Earth observation systems currently available.

I have also updated my diagram of Multispectral Earth Observation satellites with many of the systems mentioned above and a few others as well as some updates and fixes for existing data.

Multispectral Earth observation satellite sensors - September 2021

Multispectral Earth observation satellite sensors – September 2021

September 3, 2021
by chris
1 Comment

Green Marble version 2.1

I am pleased to announce an update to my global satellite image product Green Marble.

The Green Marble is a medium resolution whole earth image offering an unsurpassed uniformity in quality across the whole planet surface on both water and land coloring, based on recent data and a hundred percent cloud free.

The Caspian Sea depicted by the Green Marble 2.1

The Caspian Sea depicted by the Green Marble 2.1

The version 2.1 i introduce here is an update of the data basis used for water rendering, which is generated from Sentinel-3 OLCI data. The data basis for that has been more than doubled compared to version 2, meaning that more than 100TB of Sentinel-3 data were downloaded and processed for the production of this image.

This broader data basis resulted in a significant reduction in artefacts and noise levels in water surface rendering, which demonstrates the excellent convergence characteristics of the processing methods used.

Contrast enhanced visualization of the noise level reduction in depicting open ocean areas

Contrast enhanced visualization of the noise level reduction in depicting open ocean areas

To give some idea of the volume of data processed – here a visualization of the number of ultimately useful data samples (likely water, likely free of cloud, sea ice and sun glint) that went into processing.

Number of usable samples of water color data processed in production of the Green Marble 2.1

Number of usable samples of water color data processed in production of the Green Marble 2.1

Here a few examples of comparison between version 2 and version 2.1 illustrating the improvement in quality.


North Sea example (large: GM 2/GM 2.1)


Indonesia example (large: GM 2/GM 2.1)


Korea example (large: GM 2/GM 2.1)


South Sandwich Islands example (large: GM 2/GM 2.1)

The reduced sea ice coverage in the new image is the result of some adjustments made to processing and the additional water data used and does not reflect a reduction in actual sea ice coverage during the past years.

I updated the preview map on maps.imagico.de where you can look at the image in more detail. The product page on services.imagico.de is updated to the new version as well. If you are interested in using this image in your own project you can contact me through the form on the product page or via email. Here some additional samples featuring the improved water rendering.

Torres Strait

Torres Strait

Great Barrier Reef

Great Barrier Reef

Svalbard

Svalbard

Caribbean Sea

Caribbean Sea

August 31, 2021
by chris
2 Comments

Testing client side map renderers on polygons

I have in the past commented critically on the vector tiles hype that seems to engulf and dominate much of the digital cartography sector these days. What i dread in particular is the really saddening effect this trend seems to have in some cases on cartographic ambition of map producers – the most iconic case example being maybe swisstopo – whose traditional topographic maps are by many considered one of the pinnacles of high quality map design with uncompromising cartographic standards and ambitions – and which now sinks to the level of presenting something as crude as this as a serious contribution to their digital services.

One of the core issues of this trend in my eyes has been that it has been fueled largely by economic factors (in other words: cost cutting of map providers) while practically across the board being executed on the engineering level in an highly superficial and sloppy way. If you look at software development projects in the domain you can almost universally see the short term economic goals in the foreground and a lack of capacity and/or willingness to thoroughly analyze the task from ground up and design and develop a solution with a decent level of genericity and scalability and to invest the time and resources a decent solution calls for.

But while i am confident that this critique is justified i realize that for many people working within the vector tiles hype so to speak it is hard to follow these critical considerations without specific examples. And i also want to better understand myself what my mostly intuitive critical view of the vector tiles trend and the technological developments around it is actually caused by and better identify and where possible quantify the deficits.

Starting at the basics

What i will therefore try to do is exactly what software engineers in the domain seem to do much too rarely: Look at the matter on a very fundamental level. Of course i anticipate that there are those who say: These tools are meant to render real world maps and not abstract tests. Why should we care about such? Well – as the saying goes: You have to crawl before you can learn to walk. To understand how renderers perform in rendering real world maps you need to understand how they work in principle and for that looking at abstract tests is very useful. That you rarely see abstract tests like the ones shown here being used and discussed for map rendering engines in itself is indicative of a lack of rigorous engineering in the field IMO.

And i will start at the end of the data processing and rendering toolchain – the actual rendering – because this is where the rubber meets the road so to speak. We are talking about map rendering here which ultimately always means generating a rasterized visual representation of the map. And despite some people maybe believing that the vector tiles trend removes that part of the equation it of course does not.

In this blog post i will look at the very fundamental basics of what is being rendered by map renderers, that is polygons. The most frequent visual elements in maps that rendering engines have to render are polygons, lines, point symbols and text. And a line of a defined width is nothing different from a long and narrow polygon. Text contains curved shapes but they can with controllable accuracy be converted into polygons. Not all renderers do this in practical implementation, some have distinct low level rendering techniques for non-polygon features and there are of course also desirable features of renderers that do not fit into this scheme. So i am not saying that polygon rendering ability is the only thing that matters for a map renderer but it is definitely the most basic and fundamental feature.

As you can see if you scroll down i only look at simple black polygons on white background examples and ignore the whole domain of color rendering for the moment (color management in map renderers or the lack of it is another sad story to write about another day). Because of the high contrast this kind of test very well brings out the nuances in rendering quality.

The difficulties of actually testing the renderers

I am looking specifically at client side map renderers – that is renderers designed to be used for rendering online maps on the device of the user (the client). This creates several difficulties:

  • They typically run in the web browser which is not exactly a well defined environment. Testing things on both Chromium and Firefox on Linux i observed noticeable differences between the two – which however do not significantly affect my overall conclusions. The samples shown are all from Chromium. I will assume that the variability in results to other browsers and other platforms are similar. Needless to say that if the variability in results due to the client environment is strong that is a problem for practical use of these renderers on its own. After all as a map designer i need predictable results to design for.
  • They are only the final part of an elaborate and specialized chain of data processing designed for cost efficiency which makes it fairly tricky to analyze the renderer performance in isolation without this being distorted by deficits and limitations in the rest of the toolchain.

I managed to address the second point by running the first tests based on GeoJSON files rather than actual vector tiles. All three renderers evaluated support GeoJSON input data and this way you can reliably avoid influences of lossy data compression affecting evaluation of the rendering performance (which is – as i will explain later – a serious problem when practically using these renderers with actual vector tiles).

The test data i created, which you can see in the samples below, is designed to allow evaluating the rendering performance while rendering polygons.

Test patterns used for evaluating polygon rendering

Test patterns used for evaluating polygon rendering

I will quickly explain the different components

  1. Adjacent polygon test with four touching rectangles that are a single multipolygon geometry (which is technically invalid – but all renderers accept this). At two different rotations.
  2. Adjacent polygon test with four touching rectangles that are separate geometries. This is the classic case where AGG produces artefacts (as explained in detail elsewhere).
  3. A Siemens star like test pattern showing resolution, aliasing and color mixing biases.
  4. Dent pattern to show lines of various width both positive (black on white) and negative (white on black) features. Shows any bias between those two cases, artefacts in narrow feature rendering and step response at a strait polygon edge in general.
  5. Inverse Siemens star like pattern with narrow gaps of variable width testing geometric accuracy in rendering and geometry discretization artefacts.
  6. Test for rendering uniformity on circular arcs. Contains both a narrow black line as well as a white gap of same size for checking equal weight of those.
  7. Fine comb gradients testing color mixing and aliasing artefacts.
  8. Fan comb for testing aliasing artefacts.

The test subject

I tested the following client side renderers

  • Maplibre GL – the FOSS fork of Mapbox GL, which was essentially the first browser based vector tiles renderer and can probably be considered the market leader today (though with the fork and development lines diverging and becoming incompatible at some point this could change).
  • Tangram – originally a project of Mapzen this is the other WebGL renderer that has significant use. Having been started by Mapzen its orientation was a bit less narrowly focused on short term commercial needs but it is essentially a ‘me too’ response of Mapzen to Mapbox without much own ambition in innovation (which was a bit of a general problem of Mapzen).
  • Openlayers – the veteran of web map libraries took a very different approach and is now a serious contender in the field of client side map renderers. It uses Canvas rather than WebGL.

For comparison i also show classic Mapnik rendering. This is essentially meant to represent all AGG based renderers (so also Mapserver).

All examples are rendered in a size of 512×512 pixel based on GeoJSON data (in case of the client side renderers) or from a PostGIS database in case of Mapnik.

What serves as reference is – as explained in more detail in my previous discussion of polygon rendering a supersampled rendering – the brute force approach to rendering – performing plain rasterization at a much higher resolution (in this case to represent all the fine details in the test pattern i chose 32767×32768 pixel – that is 64×64 supersampling). When you size that down to 512×512 pixel using plain pixel averaging you get the following.

Supersampled rendering using equal weight pixel averaging (box filter)

Supersampled rendering using equal weight pixel averaging (box filter)

In terms of general precision and edge acuity this is the gold standard. However plain averaging of the pixels is prone to aliasing artefacts as you can observe in the tests for that. Other filter functions can be used to reduce that – though always at some loss in edge acuity and potentially other artefacts. Here the results with the Mitchell filter function – which massively reduces aliasing at a minor cost in edge acuity.


Box filtered reference (left) in comparison to Mitchell filtered rendering

Test results

Here are the results rendering the test geometries in their original form (as GeoJSON data).

Mapnik


reference (left) in comparison to Mapnik rendering

Mapnik is – as you can expect – fairly close to the reference. That includes similar aliasing characteristics. On the fine comb gradient it is even better since it precisely calculates the pixel fractions covered while supersampling does a discrete approximation.

The main flaw is the coincident edge artefacts you can observe on the tests for that on the lower left. Notice however these only occur with coincident edges between different geometries. Within a single geometry (tests on the upper left) this problem does not manifest (though practically using this as a workaround does likely have performance implications). The coincident edge effect also has an influence on the narrow gaps of the inverse Siemens star on the right. In other words: This does not depend on edges to be exactly coincident, it has an influence to some extent anywhere where several polygons partly cover a pixel.

Maplibre GL

Maplibre GL is interesting in the way that it has an antialias option which is off by default and without which the renderer produces results too ridiculous to show here. So what is shown here are the results with antialias: true.


reference (left) in comparison to Maplibre GL rendering

Even with the antialias option the results are bad. Primarily because of a massive bias in rendering enlarging all polygons by a significant fraction of a pixel. This is visible in particular on the fine comb gradients which are turned completely black and the dent pattern where the narrow gaps on the bottom are invisible while the thin dents on top are more or less indistinguishable in width.

In addition to this bias – which turns many of the test patterns non-functional – diagonal edges show significantly stronger aliasing (stair step artefacts), narrow gaps between polygons are non-continuous in some cases and fairly non-uniform on the circle test.

Tangram


reference (left) in comparison to Tangram rendering

Tangram does not have the bias Maplibre GL shows, positive and negative shapes on polygons are rendered in a matching fashion. But it shows similar levels of aliasing which produce a significant amount of high frequency noise even on strait lines and smooth edges like the circle test. Thin lines are rendered proportionally in weight but non-continuously. And the comb gradients produce massive artefacts.

OpenLayers


reference (left) in comparison to Openlayers rendering

OpenLayers is very different in its technical basis of rendering and this shows in the results. In many ways it is the closest to Mapnik and the reference rendering and it shows the least edge aliasing artefacts of all the client side renderers – and accordingly also the clearest representation of the circle test. It however also has its issues, in particular it shows the coincident edge artefacts similar to AGG based renderers (both on the separate and combined geometries), it has issues in proportionally rendering narrow polygon features – including some bias towards the dark (positive) lines compared to the bright (negative) lines.

With actual vector tiles

So far i have shown test results of vector tiles renderers not using actual vector tiles – to look decidedly at the rendering quality and precision and not at anything else. But the question is of course: Do they perform this way, for better or worse, also with actual vector tiles.

The question is a bit tricky to answer because vector tiles are an inherently lossy format of storing geometry data. Coordinates are represented as integers of limited accuracy or in other words: They are rounded to a fixed grid. In addition most vector tile generators seem to inevitably perform a lossy line simplification step before the coordinate discretization and the combination of the two tends to lead to hard to predict results.

The PostGIS ST_AsMVTGeom() function for example (which is meant to be a generic all purpose function to generate geometries for vector tiles from PostGIS data) inevitably performs an ST_Simplify() step without the option to turn that off or adjust its parameters.

So what i did first for testing if there is any difference between how GeoJSON data and vector tile data is rendered is to generate very high resolution vector tiles (extent 65536) as reference. However as it seems only Openlayers seems to actually work with this kind of high resolution vector tiles directly, both Maplibre GL and Tangram seem to reduce the resolution on the client side (the motivation for which eludes me). Hence evaluation of these renderers is limited to normal low resolution tiles.

Client side coordinate discretization in overzoomed display in Maplibre GL (left) and Tangram (middle) in comparison to Openlayers

Client side coordinate discretization in overzoomed display based on the same vector tile data in Maplibre GL (left) and Tangram (middle) in comparison to Openlayers (right)

Here are examples generated with the PostGIS ST_AsMVTGeom() default extent of 4096 – in comparison to the GeoJSON tests shown above.


Maplibre GL rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)


Tangram rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)


Openlayers rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)

As you can see the differences are small and mostly in those areas where aliasing artefacts are strong already. So the principal characteristics of the renderers are the same when used with vector tiles compared to uncompressed geometry data.

What this however also shows is that vector tiles are an inherently lossy format for geometry representation and that there is no such thing as a safe vector tiles resolution beyond which the results are indistinguishable from the original data or even where the difference does not exceed a certain level. Keep in mind that the test patterns used are primarily rendering tests meant to show rendering performance. They are not specifically intended as test patterns to evaluate lossy vector data compression. Also it seems that the resolution of 4096 steps when generating tiles is pretty much on the high side in practical use of vector tiles and the level of lossy compression using ST_Simplify() performed by ST_AsMVTGeom() is less aggressive than what is typically applied.

I will not go into more detail analyzing vector tiles generation here and how rendering quality is affected by lossy compression of data. Doing so would require having a decent way to separate coordinate discretization from line simplification in the process.

Conclusions

Doing this analysis helped me much better understand some of the quality issues i see in vector tile based maps and what contribution the actual renderer plays in those. With both the lossy data compression in the tiles and the limitations of the rendering playing together it is often hard to practically see where the core issue is when you observe problems in such maps. I hope my selective anaylsis of the actual rendering helps making this clearer also for others – and might create an incentive for you for taking a more in depth look at such subjects yourselves.

In terms of the quality criteria i looked at here, that is primarily precision, resolution and aliasing artefacts in polygon rendering, the tested client side renderers perform somewhere between so-and-so and badly. Openlayers shows the overall best performance. Tangram is on the second place, primarily because of much more noisy results due to aliasing artefacts. Maplibre GL makes the bottom end with a massive bias expanding polygons beyond their actual shape essentially rendering many of the tests useless and making any kind of precision rendering impossible – while being subject to similar levels of aliasing as Tangram.

Do i have a recommendation based on these results? Not really. It would be a bit unrealistic to make an overall recommendation based on a very selective analysis like this. Based on my results i would definitely like to see more examples of practical cartography based on the Openlayers vector tiles rendering engine.

If i should really give a recommendation to people looking to start a map design project and wondering what renderer to choose for that it would be a big warning if you have any kind of ambition regarding cartographic and visual quality of the results to think hard about choosing any of the client side renderers discussed here. Because one thing can be clearly concluded from these evaluations: If we compare these renderers with what is the state of the art in computer graphics these days – both in principle (as illustrated by the reference examples above) and practically (as illustrated by the Mapnik rendering – though it is a stretch to call this state of the art probably) – there is simply no comparison. Is this gap between what is possible technically and what is practically available in client side renderers warranted by engineering constraints (like the resources available on the client)? I doubt it, in particular considering what is behind in the web browser and how resource hungry these renderers are in practical use. But i can’t really tell for sure based on my limited insights into the domain.

July 12, 2021
by chris
4 Comments

State of the Map 2021 – observations

The last weekend the State of the Map conference 2021 took place – the second time as a purely virtual conference, you can find my comments on last year on this blog as well.

This commentary is somewhat preliminary because i did not yet view some of the most interesting talks because of real world obligations and the multiple competing tracks. And since the talks were pre-annouced to be recorded and to be made available afterwards while the panels, workshops and self organized sessions were not i focused on the latter a bit more.

The concept of a virtual conference – second iteration

I presented some general musings on the idea and appeal of a virtual conference in last year’s comments most of which still applies of course. Many of my more radical suggestions to depart from a pure 1:1 virtualization of an in-place conference and embracing the fundamentally new chances of the virtual format have not been followed. I have the impression that many of the influential people in the OSMF and around the conference still view the virtual conference as a temporarily necessary stop-gap replacement of what they actually want – a physical meeting of the international OSM jetset – and therefore don’t want to start things that could in any way be incompatible with moving fully back to what they know and enjoy.

Compared to last year where everything had a fairly improvised feel to it, this year the whole conference framework was much more slick and organized. But to me this is not really a positive thing. It is not that last year things technically did not work on a larger scale and this year i saw plenty of cases of (minor) glitches, in particular when you tried to join a room the browser tab often froze for a considerable amount of time. To me this slickness of this year’s conference was most visible in form of the much tighter constraints, less freedom, more builtin distance between the participants of the conference and less freedom for free form interaction. Specifically:

  • I missed the pads that were last year used for comments, questions and discussions of the talks. The free form and non-linear nature of the pads offered so much more freedom for interaction, working together and expressing yourself than this year’s linear chats and isolated fire-and-forget questions.
  • Linear group chats in general pose a structural disadvantage to slow typers and non-native English speakers and incentivize asocial tactics to dominate a discussion. They promote shallowness in discourse and essentially prevent any more elaborate discussion with multiple participants.
  • Last year’s SotM was essentially an open event where everything (streaming of talks, interactive conference rooms, pads for discussion) was openly accessible to everyone without registration. I saw this as a huge benefit and it contributed a lot to the open get-together flair of that event. This year everything (except for maybe streaming of talks) was only open to registered visitors of the conference (and i am not even sure you could still spontaneously register during the conference).
  • There does not seem to be a permanent record of the chats available to follow up on and continue afterwards.

Don’t get me wrong, this was still overall a well made conference and i saw a lot of effort on side of the organizers to make this suitable and inviting for a large variety of people (including in particular the translation efforts – which i did not check out myself). But from my perspective it was in many ways also a step back compared to last year and i kind of fear that is in a way a (potentially mostly subconscious) step to pave the way back to a primarily non-virtual conference again.

Observing filter bubble maintenance in action

The most memorable impression for me was observing the echo chambers most of the panels at this conference turned out to be and essentially observing filter bubble maintenance in action.

The panels i am talking about here are:

First of all there might be a bit of a cultural difference of course. To me a panel discussion on any serious topic would be an event where people with significant differences in their perspective and opinion on a matter meet, each presenting their views and then discussing them so they come to insights that represent an additional value compared to the sum of its parts. In other words – to me a panel discussion is essentially a dialectic format.

The panels from this conference however were more like the panels of a fan convention where actors and producers of popular performing arts praise themselves and each other for their achievements, tell anecdotes and engage in friendly and shallow banter and the main aim is for everyone to have a good time and enjoy themselves.

Now don’t get me wrong – my issue is not that such events took place at SotM. What i found remarkable however is how the participants of all of these panels clearly thought that they were having a meaningful and insightful conversation while what happened was essentially that they reaffirmed each others’ existing preconceptions. The main result of the events was the reinforcement of the existing filter bubbles of the participants and the parts of the audience on the same wavelength by excluding huge parts of what is already a small and relatively homogeneous subset of the OSM community (the conference visitors) from their perception and celebrating a friendly conversation within that bubble to be a discussion with meaning regarding and representative for the larger reality.

All of these panels had participants that could have provided very valuable contributions to a dialectic or argumentative panel discussion on the subjects discussed (and where i would for example definitely say i could learn a lot from having a conversation with them on the subject in question). But without anyone on these panels questioning the preconceptions and assumptions of the participants and opening their eyes to other perspectives on the matter the intellectual value and the gain in insight of these events for listeners on the subject matter was negligible.

In the Paid-Editing/AI-Mapping Tools panel – which was moderated by a professional scientist who should by profession be used to and experienced in questioning and challenging his views and preconceptions i decided to inquire about this with the question:

Why does your panel consist exclusively of people working for large corporations. Do you think others (small businesses, self employed people, hobbyists) have nothing important to contribute on the matter of paid editing and AI mapping tools or did those you have asked decline to participate?

The answer was sobering: They indicated not to believe that those not represented on the panel in their eyes had nothing important to contribute but they still chose to only invite representatives of large corporations.

Some thoughts on the map reading Q&A

When the call for proposals for this conference was announced i was fairly unsure if to submit something. The most natural thing to submit would have been a talk. But the talks were – like last year – to be pre-recorded to then be streamed to the visitors at the specific times of the schedule only. It seems a bit weird to me – not to say disrespectful to the international audience residing in different time zones – to impose such an artificial constraint (since the talks are prerecorded) and to essentially imitate a live event without it actually being live.

Now having talks pre-recorded and not live makes a lot of sense, in particular it lowers the hurdles to have a talk at such a conference massively. Last year i suggested to make use of the natural possibilities coming with that and to make the talks available to be watched by the visitors whenever is convenient for them in advance and to then asynchronously discuss them and to wrap that up with a live discussion similar to what we had now. But this suggestion was not picked up, probably because it was perceived to be a too radical change and it would be incompatible to moving back to a live non-virtual conference without giving that up again of course.

Anyway – my thought was if the conference organizers want to have the conference as a live event only i would want to do something that actually uses the benefits of live interaction with the audience. At the same time i wanted to try introducing the idea of asynchronous participation (like in my suggestion of making the talks available for viewing in advance) into what i do. That is how i developed the idea of an interactive Q&A workshop.

Here are my spontaneous observations and impression of how it went:

For the pre-event asynchronous involvement of the audience i had created a wiki page and a pad to submit topics to discuss. Speaking live at a virtual event of this form is difficult because there is an almost complete lack of real time feedback from the audience. At the same time it is live so in contrast to a pre-recorded talk you can’t just make cuts and re-record certain parts or the whole thing if necessary. Because i kind of anticipated that my original plan was there to be more audience interaction during the workshop.

But while i had advertised it on the diaries and on the OSM-Carto issue tracker there was only one submission (From Jerry (SK53), many thanks for that) before the start of the conference. And Jerry was not present live in the workshop (which was fine – turning up at the workshop was not a prerequisite for submitting case examples) so i could not start off with an interactive approach to the whole thing. And although during the conference a number of more submissions came in i ended up not having any live dialog in audio/video with the audience. That was a bit unfortunate (and i could probably have tried to encourage that more) – it would likely have been an improvement for both me and the audience to connect a voice and a face to the questions and have more symmetric discussion of the case examples.

In the end most of the interaction took place on the chat and the questions. The splitting into chat and questions in the conference interface was something really confusing and IMO quite pointless. You could only show either the chat or the questions and it is easy to miss additions to one of them while concentrating on the other. I see that there is a potential benefit of having anonymous questions but even that would seem possible while still conflating the two in the same display.

Over the duration of the workshop (which ended up to last for 1:30 instead of just one hour) the interactivity improved (probably mostly because i got more used to keeping an eye on the chat and questions and integrating them into my discussion) but for my taste it was still too asymmetric – as said i would have liked more true dialog.

The other important observation i made: For a live event like that involving a screenshare presentation it would be really good to have a triple-screen-setup. You would essentially need:

  • one screen with the conference interface (video, chat, controls etc.)
  • one screen to share
  • one screen for your notes and staging of materials to share

I only had two screens set up connected (had a third one – but that was on a different computer so of limited use) which made things a bit more complicated.

In terms of the actual content of the workshop – it ended up being a bit less basic map reading and more advanced map design questions – like problems of scale dependent rendering and how to deal with the rural-urban divide. I had prepared some material on the matters of road layering and urban landuse, which was more focused on basic explanation of the map but which i ended up not using because there were more than enough questions from the audience to fill the time. But i am a bit afraid that for some of the less experienced listeners the whole workshop was a bit too advanced towards the end.

What i am kind of wondering is if it would make sense to develop something like a regular recurring event out of this outside SotM. Considering how widely the standard map style is used and referred to in the OpenStreetMap world it is quite definitely severely under-explained and under-discussed on a level above the basic “I want feature x to be rendered” and outside the scope of actual style development. Any input on that idea would be welcome.

Navigating the Maze

May 12, 2021
by chris
10 Comments

Navigating the Maze – part 2

This is the second part of a two part post on roads rendering in OpenStreetMap based digital rule based map styles. In the first part i have explained and analyzed the roads rendering system of OSM-Carto and its issues, here i want to explain and discuss the re-designed road rendering system i drafted for the alternative-colors style and the improvements and new features i implemented in it.

A big warning upfront: This road rendering framework is essentially just a sketch. I will explain in the end what i hope in the long term eventually will come out of it. At the moment this is – as said – a bit sketchy and i don’t want to make any claims that it is overall better by any metric than the existing one in OSM-Carto. Or in other words: It is as much a study on what is possible as much as it is one for what is not. Keep that in mind when you think about actually using this. Also if you work with this technique or in general with the AC-style you should be sure to turn off JIT optimization in the Postgresql configuration. Because that does not deal well with the kind of queries used here apparently.

Rethinking layered rendering

The technical limitations of the OSM-Carto road rendering system i discussed in the first part are to some extent caused by specific design choices in OSM-Carto. But for the most part they are the result of principal limitations of the toolchain used by OSM-Carto. The most fundamental limitation of that is that the order in which you query the data from the database and how this is split into a lot of separate queries is directly tied to the order in which things are drawn on the map. There are some limited options to interfere with that on the raster compositioning level (OSM-Carto is using that a bit already and i use this more in the alternative-colors style – a topic for a different post) but in the end you are pretty fundamentally stuck with this paradigm of query order and drawing order being tied to each other.

Most of the software OSM-Carto is using – specifically Carto and Mapnik – are no more actively developed. That means there is not much of a chance that anything will change about this constraints. This also means that the only major component of the OSM-Carto toolchain that is still actively developed is Postgresql and PostGIS.

This essentially suggests the most promising path for rethinking the roads rendering is to work around the limitations of the layered rendering framework as described by combining all road rendering components into a single layer. If you look at the layer list in the first part with 16 layers you can see that this is a pretty radical step. Fortunately there are already quite a few pre-existing changes in the alternative-colors style that make this a bit simpler. Some layers i have dropped (tourism-boundary), others i had moved outside the road layers range (cliffs), which leaves the following layers to deal with:

  • tunnels
  • landuse-overlay
  • line-barriers
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing
  • highway-area-fill
  • roads-fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom
  • waterway-bridges
  • bridges

In addition i moved the aerialways layer above the bridges (because practically most aerialways will probably be physically above bridges they cross) – although this could in the future be changed and integrated back with taking the layering into account of course. That is still 13 layers of course.

So why would i want to combine all of this in a single layer? This has a number of advantages. Since one layer means one SQL query only you don’t have to query stuff several times from the database and you can avoid duplication of SQL code much better. At the same time addressing some of the fundamental limitations of the OSM-Carto road rendering, in particular the lack of proper layering for highway polygons, becomes much easier and does not require a significant volume of additional code.

And most importantly you don’t have to deal with the limitations of the various methods to define the drawing order i discussed in detail in the first part and the complex and difficult to understand interaction between them because you can simply put the features in the order you want to render them in using ORDER BY in SQL. That means you cannot use group-by and attachments any more of course and for every use of attachments you have to essentially duplicate the features to be drawn in SQL (like using a UNION ALL query). But practically that is relatively simple and adding an ORDER BY statement to the wrapping query of the single roads layer is a much more intuitive and better maintainable way to do this in my opinion.

Layering in SQL

So how does the query combining all these 13 layers and the improvements i implemented look like? Before you go to looking through the more than 1700 lines of SQL code you can have a look at the basic structure where i stripped most of the actual queries, in particular the lengthy column lists which make up most of the query. A more elaborate sketch of the structure is available as well. The query makes fairly extensive use of common table expressions to query different sets of features and then use them in different contexts. On the base level of the query (roads_sublayers) we have the CTEs

  • roads_all – which contains all the linear road features from tunnels/bridges/roads-casing/roads-fill as well as things derived from them (for new features – i will discuss those later),
  • road_areas_all – which contains all the road polygon features from highway-area-casing/highway-area-fill
  • tc_all – which contains the stuff from turning-circle-casing and turning-circle-fill

These CTEs currently are independent of each other. roads_all basically contains the UNION ALL from the existing road layers combining highway lines and railway lines – plus the alternative-colors specific features, which use a number of additional CTEs that reference each other.

Based on the three main CTEs there is a stack of 16 sublayer queries combined with UNION ALL. You can find the non-road layers from the above list in there while the road layers do not match the legacy layers directly. Ignoring for the moment those sublayers connected to new alternative-colors specific features we have:

casing
background
fill
area_casing
area_fill
tc_casing
tc_fill
waterway_bridges_casing
waterway_bridges_fill
line_barriers
ferry_routes
landuse_overlay

We don’t need separate sublayers for tunnels and bridges because the sublayers are not what primarily defines the drawing order. The drawing order is defined by the ORDER BY statement that follows the sublayer stack.

That means across all sublayers with a non-null int_bridge attribute (that is everything except the non-roads layers) the features with int_bridge=yes are rendered on top in the order of their layer tag. It does not matter what these are, whether road line bridges, highway polygons, turning circles or waterways. Same for tunnels at the bottom. Apart from that the ordering is similar to the one in OSM-Carto – although i made some adjustments, the effects of which are however mostly very subtle.

Map design in SQL

The most important result of this change is that it allows consistently layering all bridge and tunnel features – no matter if linear roads, highway polygons or waterway bridges – according to the layer attribute as explained in the first part. Here the layering in comparison between OSM-Carto and the alternative-colors style:

Linear roads and waterway bridges layering in OSM-Carto (left) and AC-Style (right)

Highway polygons layering in OSM-Carto (left) and AC-Style (right)

The other fundamental thing i changed and that was also an important prerequisite for the new features i added is to define the line widths for drawing in SQL. This has a two-fold purpose: First it is to make these settings available in SQL for geometry processing and second: To avoid the error prone repetitive way of defining the drawing widths for the different zoom levels as shown in part 1. Parts of this change were already implemented in previous changes where i introduced implicit embankment rendering. This is currently based on a YAML file where you define the different line widths. This file is processed by a python script into alternatively MSS code or an SQL function containing nested CASE statements for selecting the line width based on zoom level and road type. This method is not yet set in stone – an alternative would be to have a separate table in the database containing the line width values. I have now extended approach to also include the casing width values in addition to the overall line width.

Ground unit line width rendering

One thing this change facilitates is allowing to more easily implement ground unit line width rendering at the high zoom levels. The motivation for that is the following: At the low zoom levels roads are universally rendered with a line width larger than their real physical width on the ground. That is universally the case in maps because otherwise the roads would not be visible at smaller scales and even when they would be that would not make for a well readable map. There is however a transit at some point where the drawing width or the roads becomes less than the physical width of the roads. That transit typically happens – depending on latitude, road class and the specific road – at zoom levels between 16 and 18. Drawing the roads more narrow than they are physically is however typically not of benefit for map readability and it also conflicts with the general aim you have when designing large scale maps to move to less geometric abstraction and a depiction closer to the geometric reality on the ground.

Unfortunately you cannot address this simply by increasing the nominal drawing width of the roads in general because not all individual roads of a certain class have the same width of course and what you definitely don’t want at the high zoom levels is to draw the roads wider than their actual width and then obscure other content of the map next to the roads. And fortunately quite a few of the roads in OpenStreetMap have a physical width specified – two million (or about 1 percent) and even more (12 million or 6 percent) have a lanes tag that allows estimating the likely physical width of the road with some accuracy.

To be able to draw lines in ground units on the map you have to be able to convert between map units (also called Mercator meters), ground meters and pixels on the map. The conversion factor between ground units (as tagged with the width tag for example or as estimated from the number of lanes) and map units is called the scale factor of the projection and unfortunately there is no support for directly getting that in PostGIS. I used a generic function that allows approximating that:

ST_DistanceSphere(
  ST_Transform(ST_Translate(geom, 0, 1), 4326),
  ST_Transform(geom, 4326)
)::numeric from (select ST_Centroid(!bbox!) as geom) as p

I do this based on the bounding box of the metatile – not so much for performance reasons but because that ensures geometry processing within a metatile is using the same scale factor – which is essential in some cases. At the zoom levels in question the scale factor will never vary significantly across a metatile. One thing to consider of course in the future is to – for Mercator maps – replace this generic function with the simpler formula specifically for the Mercator projection (because that function is used quite a lot).

The conversion between map units and pixels on the map by the way is the factor !scale_denominator!*0.001*0.28 – so in total you have

pixels = ground_meters/(scale_factor(!bbox!)*!scale_denominator!*0.001*0.28)

This formula is used in carto_highway_line_width_mapped() – which is implemented in roads.sql. Overall the rule for the line width of roads is as follows:

  • If a width is tagged use that as the ground width.
  • If no width is tagged but the number of lanes is tagged then estimate the overall ground width based on the road class and (conservatively estimated) the typical lane width for that type of road.
  • If the ground width determined that way is larger than the nominal line width set by the style than draw it in that width, otherwise in the nominal width.

Currently the style still contains double drawing of the roads in both nominal and ground width (essentially determining the maximum graphically) because i initially had them rendered in different styling. But that should be removed in a future version.

Here is how this looks like at z17 with various different tagged width values – in comparison for different latitudes to show the effect of the variable scale.

Ground unit rendering of roads at z17 at 40 degrees (left) and at 60 degrees latitude (right)

This applies for the roads drawn with the standard casing + fill style. For airport runways the formula is varied to estimate the likely width based on the runway length (using a formula initially suggested for OSM-Carto). For road features with a simple solid/dashed line signature (paths/tracks) the ground width is only visualized at z18 and above – using the styling of footway and track polygons (with some adjustment for tracks to avoid the appearance being too heavy) in addition to the centerline signature. And this ground unit visualization is only used when the drawing width in ground units is large enough to be meaningfully depicted in addition to the centerline.

The ground unit rendering of roads however is not technically as simple as it might seem because of another implicit assumption made by the roads rendering system in OSM-Carto, namely that the drawing order of road classes with the same layer as defined by z_order is in sync with the drawing width progression. That is of course no more universally the case when the tagged width is depicted which would lead to artefacts like the following.

Road junction with artefacts

Solving this requires modifying the road geometries in those cases which is kind of a hassle to do.

Road junction without artefacts

Lanes visualization

Since i already made use of the lanes tagging for estimating the ground unit width of roads i decided to visualize the number of lanes at the high zoom levels as well. As you can see this follows the implicit rules documented which say that trunk roads and motorways are assumed by default to be two lanes. lane_markings=no is indicated with dashed lines instead (click on the image to see).

Lanes visualization at z18+

Lanes visualization at z18+ (click to see also the unmarked version)

In practical use cases ground unit line width rendering and lanes visualization looks like this:

Roads in Strasbourg at z18

Roads in Heidelberg at z18

Roads in Freiburg at z19

This one demonstrating the unfortunate and semantically nonsensical practice of German mappers to tag every road under a bridge as a tunnel.

Of course this style of visualization for both the lanes and the tagged width of the road does not work too well when the number of lanes and the road width changes. This requires mapping and interpretation of additional data, in particular what is mapped with placement=*, which is currently not taken into account.

Turning circles & co.

While we are on the matter of drawing width or road lines – i also extended the existing visualization of turning circles and mini-roundabouts from OSM-Carto with distinct symbolization of both of these as well as turning loops and passing places.

turning circles, turning loops, mini-roundabouts and passing places at z17

turning circles, turning loops, mini-roundabouts and passing places at z17

All of course working in combination with the bridge layering and variable width depiction. The diameter tag for turning circles is also taken into account.

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads ...

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads …

... as well as tracks

… as well as tracks

Sidewalks

The next new feature i looked into visualizing is implicitly mapped sidewalks and cycle tracks. Like in case of embankments sidewalks can be mapped both implicitly as an additional tag on the road (sidewalk=both/left/right) as well as explicitly separately as a way with highway=footway and footway=sidewalk. Both are widely used and there is no clear preference among mappers and it is therefore a bit problematic that OSM-Carto only renders the explicit sidewalks. Like in case of embankments the explicit version is easier to render in a primitive fashion while good quality visualization is much easier with the implicit mapping because you can more easily take the relationship between the road and its sidewalk into account.

What i implemented is a demonstration how visualization of implicit sidewalk mapping as well as that of implicit cycleway tracks with cycleway:right=track/cycleway:left=track can work. This is shown at z18 and above

Rendering of sidewalks and cycleway tracks

Rendering of sidewalks and cycleway tracks

and works also in combination with bridges, fords and implicit embankments. They are not rendered on tunnels to avoid obscuring ground level features.

Rendering of sidewalks in combination with various road variants

Rendering of sidewalks in combination with various road variants

And here is how this looks like in real world situations.

Sidewalks in Ladenburg at z18

Sidewalks in Ladenburg at z18

Sidewalks in Portland at z18

Sidewalks in Portland at z18

Steps details

Rendering of roads in their ground unit width also provides more space to visualize additional details on steps. The normal drawing width of steps does not really provide enough room for showing more information without affecting the recognizably of the steps signature. Here is what is shown with nominal drawing width and when the drawing width is large enough at z18+ for wider steps tagged with their width.

Steps details at z18+

Steps details at z18+

Unpaved roads

The distinct depiction of paved and unpaved roads is one of the oldest open issues in OSM-Carto. The difficulty is that it has to work in combination with a lot of other road properties already visualized. Various forms of dashing have been tried and have been found either to be very noisy in appearance or too similar to existing depictions of tunnels (which use a dashed casing), construction roads (which use a dashed fill) or access restrictions (which use a dashed centerline on the fill) to be intuitively understandable without affecting map readability in general. The most promising idea so far has been the use of a fine grained structure pattern for the road fill. That is highly intuitive in its meaning and can be tuned to be fairly subtle. Lukas Sommer has developed this idea some time ago to be ready for use but unfortunately Mapnik did not support 2d fill patterns for line features and while this was added as a feature some time ago this is not yet available in many production environment and especially not in release versions of Kosmtik. And workarounds for this are a bit ugly to implement in the OSM-Carto road rendering system. I have now added this to the AC-style by generating polygon geometries for the unpaved roads using ST_Buffer() and rendering those instead of the normal line features. That is easily possible with the new setup since the line widths are available on SQL level anyway.

The basic styling of the unpaved road rendering used

The basic styling of the unpaved road rendering used – click to see at z18 with and without tagged width visualization

It is important to consider how this looks like in combination with various other road properties – which you can see in the following illustration. There are some issues with the readability of the access restrictions – which is mostly due to their depiction offering very different levels of contrast on the different fill colors. That needs a closer look at and either tuning the access restriction color or revisiting the principal design of this feature in general.

Unpaved road styling in combination with other road attributes at z17

Unpaved road styling in combination with other road attributes at z17 – click to see more

Also here a few examples of practical situations with this styling.

Unpaved roads

Unpaved roads

Unpaved pedestrian area

Unpaved pedestrian area

Open ends

That’s it for the main new road features and improvements i introduced within the new road rendering framework. There a quire a few other tweaks improving consistency in rendering – like getting tunnels and bridges rendered for raceways. And a subtle centerline on runways and taxiways. And as already hinted at there are quite a few open ends in the whole thing design wise that could use further work. But this blog post is already longer than it should be so i will close for now.

Performance of this new roads rendering approach in terms of computing resources required is not bad by the way. While the roads query is often the slowest query of the style – which is not that astonishing because it contains the data previously processed by a lot of different queries – it is not exceedingly slower than other queries. And it is often faster than the water barriers query, which has been the slowest so far and which is why i added an option to disable that for faster rendering.

Technically one important thing left unfinished is that there is currently a lot of duplication of rendering rules between SQL and MSS code. Like for example i have introduced several zoom level selectors in SQL to limit certain queries to specific zoom level ranges (of the form WHERE z(!scale_denominator!) >= 18) which are unnecessarily mirrored by a matching selector in MSS. This is redundant and prone to errors so it is important to come up with some clear principles which parts of the style logic should be in SQL and which in MSS code.

I updated the sample rendering gallery to the style changes by the way and also added quite a few new samples there.

Conclusions on part 2

The main outcome of the whole endeavor of re-designing the road rendering system this way from my perspective is that

  1. yes, there are much better ways to structure rendering of the roads in an OpenStreetMap based map style without taking shortcuts and as a result failing to consistently representing the road topology than the classic OSM-Carto road rendering setup.
  2. The ad hoc implementation of this in the way presented here based on the Carto+Mapnik toolchain is anything but ideal. In particular the difficulty of debugging the SQL code is a major concern.
  3. The map rendering framework that provides a decent platform for implementing the kind of rendering of roads presented here quite clearly still needs to be developed.

My main hope is that what i show here provides valuable additional insights into what exactly the key factors are to be able to develop map design in an efficient, flexible and scalable way (and scalable here is meant from the perspective of style development, not from the side of rendering efficiency – which is an important matter too of course – though pretty much a separate issue). At least to me it seems after working on this project i now have a much better idea of what the core requirements would be for a future map rendering and design system suitable for sophisticated and high quality rule based cartography with zoom level/map scale and styling specific geometry processing.

In terms of the new features implemented it is too early to draw any definitive conclusions i think. Most of them, in particular the variable width ground unit rendering and the lanes depiction i consider to be in a highly experimental state with a lot of open issues – both technically and design wise. The question of what constitutes the core of meaningful OpenStreetMap based cartography for the very high zoom levels (z19+) that goes beyond simple extrapolation of lower zoom level designs plus functioning as a POI symbol graveyard and that does not depend on mappers essentially directly hand drawing the map is still largely undiscussed.

Navigating the Maze

May 11, 2021
by chris
1 Comment

Navigating the Maze – part 1

This post is part of a longer series on map design questions. I have not written much more recently on my work on the alternative-colors style and there is a bit of a backlog of topics that deserve some discussion. I will make a start here with the most heavy of all – the roads. In this first part i am going to explain the OSM-Carto road rendering system and analyze its issues and limitations.

The challenge of depicting roads

The without doubt most complicated part and for any OSM-Carto map style developer the most serious challange when working on OSM-Carto or one of its derivatives is the roads layers. As Paul some time ago compactly put it the roads rendering in OSM-Carto alone is more complex than many other map styles are in total. Roads (and in context of OSM-Carto roads by the way generally refers to all lines and other features along which land navigation of humans takes place – including roads for automobile navigation, paths for navigation by foot, animal or other vehicle as well as railways and traditionally also runways and taxiways on airports, which however have more recently been moved outside of the roads layer system in OSM-Carto) can be complex in their semantics, connectivity and above all in their vertical layering. Representing that in a well readable and accurate way in a map is a challenge to put it mildly.

Most simpler map styles take shortcuts in this field – which tends to show up in map in non-trivial situations. See for example the following case example:

Road example in Dallas, US in OSM-Carto

Road example in Dallas, US in OSM-Carto

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

All maps have their flaws in roads rendering – OSM-Carto in particular with the labeling – which however is a different matter that i am not going to cover here. But beyond that it offers by far the most consistent display of the connectivity and layering of the roads in this intersection.

The way this is done in OSM-Carto i will describe in a moment – but before i want to emphasize that OSM-Carto producing a consistent rendering of how the roads are mapped in OpenStreetMap is of high importance for the quality of data in OSM. It helps ensuring that the information that is necessary for an accurate interpretation of the data is mapped and that the reality is not recorded in a distorting fashion to achieve a certain rendering result. More than in any other field of mapping in OpenStreetMap probably the rules of mapping roads have been developed in connection with how the data is used in rendering while still maintaining the principle that what is mapped and tagged represents verifiable elements of the geography and not just prescriptions how things are supposed to show up in the map. This is an accomplishment that cannot be overstated and that is important to maintain and extend in the future.

The OSM-Carto road layers

To understand the way OSM-Carto renders roads it is important to first understand how the drawing order is defined in the toolchain used by OSM-Carto (Carto and Mapnik). In decreasing priority the drawing order is defined by the following:

  1. The layers of the map style: Data that is rendered in the map is queried from the rendering database in the form of distinct and separate layers and is rendered in exactly the same order. The layer stack is defined in project.mml and starts with the landcover layers (at the very bottom) and ends with labels.
  2. The group-by parameter of the layer in question (in case it is present): The way this works is that the query of the layer is ordered primarily by one column of the results table and by specifying this as group-by parameter you tell carto/mapnik to group the features by that parameter and render them kind of like they were in separate layers.
  3. The attachments in MSS code. That is the ::foo selectors. Those kind of work as a method to accomplish layered drawing within a single layer. But in contrast to the group-by parameter every attachment needs to be coded explicitly, you cannot automatically base attachments creation on some attribute in the data.
  4. The order in which the data is queried in SQL (typically through an ORDER BY statement)
  5. The instances of the drawing rules in MSS code: Those are the prefix with a slash you put in front of the drawing rules. Like a/line-width: 2; b/line-width: 4; Such rules lead to several lines being drawing, one immediately after the other, on a per-feature basis.

OSM-Carto contains exactly two layers that use the group-by feature: The tunnels and the bridges layers for the roads. Overall the roads related layers (and other layers in between) look like this (from bottom to top):

  • tunnels (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill
  • landuse-overlay (military landuse overlay)
  • tourism-boundary (theme park and zoo outlines)
  • barriers (line barriers like fences, hedges etc.)
  • cliffs
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing: with attachment ::casing
  • highway-area-fill
  • roads-fill: with attachments ::halo (low zoom only), ::fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom: with attachment ::halo
  • waterway-bridges
  • bridges (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill

To better understand the terminology used – here an illustration for how the different parts of the road signature are called.

Road signature elements at high zoom levels (z18)

Road signature elements at high zoom levels (z18)

Road signature elements at low zoom levels (z11)

Road signature elements at low zoom levels (z11)

All the individual road line layers (tunnels, roads-casing, roads-fill and bridges) are ordered in SQL primarily by layernotnull (which is the layer tag with an implicit default value of zero assumed) and by z_order – which represents the hierarchy of road classes.

If we ignore most of the non-road layers for the moment and focus on the high zoom levels we have essentially the following drawing order:

  1. tunnels – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)
  2. casings of normal non-tunnel, non-bridge roads – ordered by
    • first turning circles, then polygons, then linear ways
    • layer
    • z_order (+ some other criteria in SQL)
  3. fills of normal non-tunnel, non-bridge roads – ordered by
    • first polygons, then linear ways, then turning circles
    • layer
    • z_order (+ some other criteria in SQL)
  4. waterway bridges – ordered by
    • layer
  5. bridges – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)

This is how this looks like visually:

Layering of linear roads in OSM-Carto

Layering of linear roads in OSM-Carto

That ordering alone however – despite its complexity – does not ensure a consistent depiction of the roads network. That also relies on the specific way the road lines are drawn. To understand this have a look at the following illustration.

Bridge drawing principles with different layers connecting

Bridge drawing principles with different layers connecting

It shows self intersecting road loops as either tunnels or bridges. On the left how this is rendered when mapped as a single way. To actually represent a road crossing over above itself in form of a bridge or tunnel rather than intersecting with itself on the same level you have to split the way into two and assign different layer values.

That the two ways still visually connect where they meet at the top with their ends depends on the way they are drawn with flat line caps for the casing and round line caps for the fill. This leads – as you can see in the rightmost version – to some smaller artefacts when the bridge/tunnel way ends meet at an angle. That brings me to technical mapping rule number one that is good to consider in mapping:

Bridge or tunnel ways connecting at their ends should always have collinear segments where they meet.

The other thing that i hope this explanation shows is the meaning of the layer tag in OpenStreetMap. The wiki is fairly detailed on this but also a bit confusing. What the layer tag does is describing the relative vertical position of the physical object the feature represents, in particular a road, relative to other elements in the immediate vicinity that it is not connected with. It does not define the drawing order in the map (which depends on how exactly the map shows different features), it does not document the vertical position of a feature relative to ground level. This leads to the second technical mapping rule you can take away from this

Use the layer tag sparsely (both in the extent of where you apply it and regarding what cases you apply it in) where it is necessary to avoid ambiguities in the description of the 3d topology of what is represented by the data.

In all other cases there are other tags that will most likely better describe what you want to document – like bridge=*, tunnel=* location=underground, level=*. Or you are trying to do something you should not do (like – as a mapper – drawing the map yourself).

Limitations of this approach

Some limitations of the described system of road drawing are already visible in the samples shown. Others are not and i will have to explain those in the following. Before i do that however i want to mention one additional aspect of the OSM-Carto road rendering system that i have not mentioned so far and that frequently is an additional source of difficulty for developers.

The actual drawing rules for the roads in CartoCSS for large parts serve the different layers together. In particular for example the styling of the roads fill for tunnels, ground level roads and bridges is using the same CartoCSS code. That has the advantage that the different styling variants (see image below) are consistently applied to all the layers. And changes made to that do not need to be duplicated at several places. But it also makes analyzing and understanding the code overall more difficult because after the road features are split into the different layers for rendering them in the proper order these layers are tied together into a meta-maze again for styling.

Different styling variants on all the road classes - many of these can be cross-combined

Different styling variants on all the road classes – many of these can be cross-combined

Beyond that the main principal technical disadvantages of the OSM-Carto road rendering system are:

  • It requires quite a lot of SQL code being duplicated into the different road layers, in particular the tagging interpretation that is being done on SQL level. Some of this has been reduced more recently using YAML node anchors and references – specifically the roads-casing and roads-fill layers as well as the turning-circle-casing and turning-circle-fill layers. That only works when the SQL code is exactly identical though.
  • This of course also means that it requires unnecessarily querying the same data several times from the database. Practically this is not a big issue performance wise but still it is not a very elegant way to do this.
  • Any non-trivial improvements and extensions to be made to the road rendering system typically require additional layers or at least adding to the SQL code that then needs to get synchronized across several layers and adding to the already significant complexity of the whole system.
  • If you look at roads.mss you can find a lot of fairly mechanical repetitions of code to implement the variation of styling parameters across zoom levels. Like the following:


    [zoom >= 13] { line-width: @secondary-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19; }

    Often this is even repeated for other layers or variants – like in this case with:

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @secondary-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @secondary-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @secondary-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @secondary-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @secondary-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @secondary-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @secondary-casing-width-z19; }

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @major-bridge-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @major-bridge-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @major-bridge-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @major-bridge-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @major-bridge-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @major-bridge-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @major-bridge-casing-width-z19; }

    That adds a lot of code volume and is prone to errors. It also makes styling changes that go beyond simple adjustments of the parameters in the CartoCSS symbols used complicated.

On the level of currently present functionality some of the most significant limitations of the current system are (and note these are only those related to the main road layer stack as described – things like road labeling are a different story):

For the polygon features rendered (that is primarily highway=pedestrian and highway=service polygon features) there is no support for bridges and tunnels. That means the above grid example with highway=pedestrian and highway=service polygon features looks like this:

Layering of highway polygons in OSM-Carto (or the lack of it)

Layering of highway polygons in OSM-Carto

Solving that would be possible but you would have to integrate the polygons into all the linear road layers and that would significantly increase their complexity (and the amount of code that needs to be duplicated between them). Same applies for turning circles obviously – although this is a much less significant limitation practically of course.

Another limitation is that while the waterway bridges are correctly layered among themselves they are not interleaved with the road bridges. So a waterway bridge is always drawn below a road bridge independent of the layer tags.

Road layering with waterway bridges in OSM-Carto

Road layering with waterway bridges in OSM-Carto

Practically this is not a big limitation of course (how many waterway bridges are there on Earth where there is a road bridge below?).

What is more important is that for avoiding complexity OSM-Carto has some features not within the road layers that should be included because they can be above or below road features in reality. Specifically that is bus guideways (which have always been separate creating various issues) and airport runways and taxiways (separated more recently) – which is rarely a problem but there are cases where it is (though in most cases of enclosed pedestrian bridges spanning taxiways the bridge is purely mapped as a building and not as a footway).

What i have not covered are the frequent concerns about overall drawing order of roads, in particular road polygons, above other map features, in particular buildings and landcover features. That is a whole other subject and one connected to much more fundamental question about map design and mapping semantics in OpenStreetMap. And it is not something related to the specific design of road layer rendering used by OSM-Carto.

Conclusions on part 1

I tried above to explain how OSM-Carto renders roads and how this system – while being one of the most sophisticated road rendering systems you can find in OSM data based maps – has its limitations. Anyone who tries to implement improvements to map rendering in the style in any way related to roads struggles with those. In the course of my work on the alternative-colors style i have now decided to try re-designing the roads rendering in a fairly fundamental way. How this looks like and what new features and improvements can be achieved with that will be subject of the second part of this post.

April 15, 2021
by chris
2 Comments

On group communication channels

What i want to discuss here a bit is channels and platforms for group communication (that is communication between more than two people) in the context of an open social project like OpenStreetMap. For many years every now and then people turn up on OSM communication channels and express dissatisfaction with either the particular channel or with communication in the OSM community in general – often connected with the suggestion that everyone should move to [fill in your favorite hype of the day platform]. I want to explain here my personal criteria what aspects qualify a communication channel or platform for productive, constructive and inclusive group communication in context of a global multicultural social project like OpenStreetMap.

A few preliminary remarks: First i want to clarify that this analysis concerns group communication in the sense of symmetric communication between more than two participants on equal levels. For one-to-one communication or asymmetric communication (one-to-many with no symmetric back-channel) different criteria apply. Second, a diverse global community like OpenStreetMap depends on there being diverse communication channels used for communication. Concentrating everything on a few channels and platforms would be an issue on its own. OpenStreetMap can only work in a sustainable way if communication and social interaction within the community does not depend on there being a central communication channel everyone engages on. The very idea of OpenStreetMap is that the main database is the element that holds the project together socially and trying to substitute that with some other means would fundamentally weaken the project.

The criteria

  1. Asynchronous communication: In a global, multilingual community like OSM there is inevitably a significant variation in how fluent people are in the language of a specific communication channel. Asynchronous communication (that is communication where there is no expectation, technical need or social pressure for a close timing between a message and reply to that message) can enormously help to level the field in that regard. Not to mention different time zones. Asynchronous communication also helps accommodating participants with intermittent internet access.
  2. Text based communication: For similar reasons text based communication is generally preferably to audio or video communication. Audio or video group communication is of course also almost universally synchronous anyway. In addition text based communication reduces the bandwidth requirements for the participants, makes any kind of filtering massively easier and especially it reduces the participation barrier because it gives participants better control over the content of their communication and the disclosure of private information.
  3. Structured communication: One fairly essential component of group communication with a larger number of participants is transparent visibility of who communicates in reply to whom. The classic way to visualize that is threaded communication. opinions on this differ a lot – some despise the threaded display of communication while others think it is essential for communication in larger groups to work properly. This has a lot to do with digital socialization of people. Those who have become familiar with digital communication without being exposed to the concept of threading often adopt a dislike of this. And with widespread tools and platform (Microsoft, Google) of email communication over the last decade or more by default hiding threading this applies to quite a lot of people. What is fairly clear however is that if a larger number of people participate actively on a communication channel without the communication being structured into threads it will often turn more and more into a synchronous communication channel because asynchronous replies to older communication is perceived to be disruptive by the most active participants of the channel. And this comes with the issues i mentioned above for the asynchronous communication.
  4. Availability of a linkable and immutable permanent public record of past communication: This is one of the most important points. For group communication to be practically meaningful for a larger community it is absolutely essential that you can refer to past communication in conversation in a way anyone can follow and that you can resolve disagreements on the exact nature of past communication transparently. This is easiest through a complete, reliable, immutable, searchable and linkable record of the full history of communication.
  5. Open platforms and communication protocols: For long term sustainability and to avoid dependency on proprietary technology providers. That means in particular the technical interfaces to participate in the communication should be documented and both legally and technically open to be used by everyone without being tied to using specific software.
  6. Practical diversity in communication clients or interfaces: Connected to the previous point – it is of high value if communication protocols are not only open in principle but if there are also different clients and interfaces available that users can choose between according to their individual needs and preferences. Apart from personal preferences and convenience this is also a matter of accessibility – in particular for people with disabilities.
  7. Practically usable flexible and robust ways of filtering communication by the individual user: This is connected to the previous two points. In case of open communication protocols and a variety of communication clients available, one of the most important features those clients tend to distinguish themselves with is filtering options for the communication. And that is of significance on two levels. First it allows people to focus their attention of what they consider important and of interest for them rather than what others consider important. And as a secondary effect it tends to have an immensely civilizing effect on the participants of a channel. If people know everyone else on a channel has the ability to choose ignoring their messages they tend to weigh more carefully what they write and take more into consideration if what they write is likely to be perceived to be of value by others. To put it bluntly: Never underestimate the pacifying effect of a killfile.
  8. Channel is open to everyone and does not require explicit invitation or other access control or discouragement from active participation. Like the required agreement to terms and rules decided on by others than by those who are actually communicating on the channel. This is again essential for diversity in participation. For many people the psychological barrier to actively contribute in a public group communication channel is quite significant so it is important not to create additional hurdles to that. Of course ultimately most, even non-commercial, operators of communication infrastructure ultimately reserve the right to control access to their infrastructure when they consider it necessary but the key here is if that is a possibility that is practically used or not.
  9. Decentralized protocols. This is the ideal solution for the previous issue but practically there are only very few group communication channels that work in a fully decentralized fashion. Historically Usenet was the iconic example of decentralized group communication and since the demise of Usenet there has so far never been any decentralized protocol for group communication that has gained an in any way comparable significance.

And how do different channels compare?

Here is how the various communication channels commonly used in the OSM community fare regarding the criteria mentioned above. I included Usenet/NNTP although his is not practically relevant both in the OSM community and outside of it because is has historically been very important in forming digital group communication culture on the internet and as you can see it also is outstandingly good at meeting my list of criteria. The reasons for the decline of Usenet and NNTP is a whole different story of course (quite an interesting one though – and one that many people have already written on so there is not that much point in touching that here).

The information on Facebook, Telegram and Whatsapp is based on hearsay and therefore not that reliable – i have never used these platforms so i can’t really provide first hand experience.

OpenStreetMap communication channels and platforms

OpenStreetMap communication channels and platforms

Footnotes from the table:

  1. help.openstreetmap.org has limited support for structured communication.
  2. The OSM wiki in principle allows structured communication and on talk pages it is established practice to thread messages. But you have to do this fully by hand and it only works if everyone does so diligently. No builtin support for that.
  3. On Usenet server operators used to expire messages so there was practically no permanent public record in the system itself. However everyone running a Usenet server could maintain a record and in the late days of Usenet Google practically served as a reference record.
  4. The forum indicates when a post has been modified after initial posting but no record of the original message is retained. Also admins on the forum can (and do) remove messages from the record (with varying frequency, depending on the channel).
  5. The diaries themselves can be edited without the history being visible but the diary comments cannot.
  6. On the wiki the edit history is openly accessible so it serves as an immutable record (though it can be difficult to use as such practically).
  7. Github has a fairly clever system of allowing edits of communication but still allowing users to transparently see if there have been changes and what has been changed. Github however also allows completely removing messages from the public record in some cases (but it is indicated at least that a message has been removed i think).
  8. While there is no alternative interface to posting changeset comments for reading there is the popular tool by Pascal Neis.
  9. Openness in participation applies to most OSM related mailing lists and forums. There are a few exceptions – specifically osmf-talk is only open for posting to OSMF members. There are thematic and regional lists and forums with culture specific rulesets you need to accept for active participation which are usually developed through consensus of those active on the individual channel (like diversity-talk, regional forums).

Conclusion

The bottom line is: Mailing lists still rule by a huge margin compared to everything else that is used today in the OSM community. But of course the criteria i used are my personal preferences and although i have explained why i see it this way there are others who view what i see as advantages as major disadvantages. And that is fine – as i pointed out in the beginning diversity in communication is essential for a diverse global community like OpenStreetMap. There are some in OpenStreetMap who dislike that and who would want to culturally homogenize the OSM community and would like to nudge everyone to adopt the same communication platforms. That is not going to work of course. If there is one field where you can be absolutely certain that people will always vote with their feet it is group communication channels.

Lookout

What is interesting is of course to look at what the future might bring. Will there be platforms and channels that can compete with the old school techniques (NNTP and mailing lists) that came out as winners in my analysis regarding the criteria i looked at? At the moment nothing is visible in that direction i think. Focus in development in group communication technology has for quite a few years been in the real time synchronous domain but as i tried to explain above this is of relatively little importance for highly multicultural use cases like we have them in OpenStreetMap. The trend in asynchronous communication has for quite some time been towards monolithic platforms where the client and its user interface are tightly interwoven with the underlying communication protocols (or in other words: Stuff developed around some fancy web interface meant to serve the fashion of the day in UI design). That is unfortunate because it tends to take many years for a highly non-homogeneous community like OpenStreetMap to truly adopt new communication channels en large. And by the time that happens such fashion of the day frameworks have often already reached the end of their short life and are out of fashion and not maintained any more.

March 22, 2021
by chris
0 comments

Satellite image aquisitions – yearly report 2020

I am a bit late this year but i wanted to continue with my tradition to publish a yearly analysis of the recordings of the two currently operating high resolution open data producing earth observation satellites recording in images in natural color – that is Landsat and Sentinel-2.

Superficially similar analysis is sometimes done by the satellite operators as well of course but in most cases this is not very detailed and in some cases even grossly inaccurate by considering redundancies in published data files as independent recordings (which they obviously are not). Here you can find the most accurate analysis of the spatial distribution of the recordings of Landsat and Sentinel-2 based on the actually published imagery.

Development of total daily published image volume in square kilometers

You can already see from the development of the area covered that for the most part USGS and ESA continued with the same pattern as previous years. The biggest anomaly more recently has been that the USGS apparently for unknown reason stopped off-nadir recordings of Landsat 8 in polar regions for about a year (2019 Northern hemisphere Summer and 2019/2020 southern hemisphere summer) but they have more or less resumed with the previous practice in 2020.

Landsat 8 2020

Sentinel-2 operations continued with the changes made in 2019 (that is extended high latitude coverage in the north at low sun positions with Sentinel-2B, some additional small islands) throughout 2020.

Sentinel-2 2020

More important is what has not changed – and that is in those fields i suggested last year already – so i repeat them here:

For both USGS and ESA:

  • make use of the capacity you have in the northern hemisphere winter to collect more than just sporadic data of the Antarctic.

For the USGS in addition:

  • finally get rid of the last gap in your land area coverage (Iony Island).
  • record off-nadir images of northern Greenland and the central Antarctic on a regular basis (and preferably make use of the flexibility you have here with the chosen path for cloud cover minimization).

And for ESA:

  • stop the non-professional and short sighted catering of special interests and record images based on clear, uniform and transparent criteria.
  • connected to that: rethink your strategy of recording extensive ocean areas around some islands while not recording other larger islands (South Orkney Islands) at all.
  • record with Sentinel-2A and Sentinel-2B the same way or explain clearly why you don’t.

And here as in past years all of the maps showing image coverage together:

year day night day pixel coverage
2014 LS8, LS7 LS8 LS8
2015 LS8, LS7 LS8 LS8
2016 LS8, LS7 LS8 LS8, S2A
2017 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2018 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2019 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2020 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)

March 16, 2021
by chris
1 Comment

How to invent new tags in 2021

Back in 2010 Jochen wrote a practical guide How to invent new tags providing some useful advise to mappers what to consider when invention new tags. Back then however the world of OpenStreetMap was very different. There were far fewer established tags than today and OpenStreetMap was still very young and most people the guidance was written for were new to OSM and to the whole concept of cartographic data collection.

Today in 2021 OpenStreetMap and the typical situation of people inventing a new tag looks very different. There are so many widely used tags that you can hardly any more invent a tag on the green field so to speak without taking into consideration already existing tags in the semantic vicinity of the tag you consider inventing. And most people who get into the position of inventing a new tag have either already been involved in OSM for quite some time and have developed a firm opinion on how things should be mapped and tagged (not always matching how things are actually mapped and tagged) or come from outside of OpenStreetMap and have a firm opinion on how cartographic data collection should look like shaped outside of OpenStreetMap.

Therefore i decided to formulate a new guidance focusing on things that i consider practically relevant when inventing new tags nowadays. This is mostly based on years of observing and studying tags and tag use as an OSM-Carto maintainer and my observations what decides on if a tag becomes a meaningful tag being adopted and consistently used by mappers and also on observing countless tagging proposals over the years. It is not meant to replace Jochen’s guidance which focuses a lot on formal and technical aspects which i will not cover here but to supplement it with updated guidance on semantic, social, cultural and geographic considerations.

Design tags to be practically verifiable

Verifiability is the core of mapping and tagging in OpenStreetMap. While we have a number of poorly verifiable legacy tags in OpenStreetMap (like tracktype) it is a fairly universal rule that newly invented tags that are not practically verifiable fail to become successful tags. They either fail to become adopted by mappers at all (like view) or worse: they fail to develop a well defined meaning in practical use despite being widely used and therefore lead to mappers wasting time to tag information that is practically useless because it is not consistent.

Practical verifiability means that a mapper needs to be realistically able to determine locally on the ground and independent of other mappers or external data sources if a tag is correct or not. In some cases tagging ideas are inherently subjective (like the usability of a road with a certain class of vehicle). Sometimes non-verifiable tags however also result from being invented as a subjective aggregate of several individually verifiable aspects which could individually be tagged in a verifiable fashion (like to some extent natural=desert).

Another distinct class of non-verifiable tags is data that is defined not through something locally observable but through authoritative information from outside of OSM. This for example applies to ratings or restaurants or other establishments. Such data simply does not belong in OpenStreetMap.

One important aspect of verifiability is the practical ability to localize features (and to delineate them when mapping with linear ways or polygons) in a verifiable fashion. Something might verifiably exist but could still not be practically mappable in OpenStreetMap because it is not practically localizable.

Do not invent tags for storing redundant or computable information

Sometimes tags are invented where mappers are supposed to tag information derived from other data in OSM or outside of OSM or that duplicates existing data in an aggregated fashion. Examples are the tagging of the length of linear features or of the prominence of peaks.

Such tags – as convenient as they might seem to data users – are a bad idea because they take valuable time of mappers to compute and aggregate information that could also be automatically computed and there is nothing to ensure that the information is kept up-to-date so it will never be reliable so data users who want to have this kind of information in good quality will always try to compute it themselves.

Be mindful of existing tags when designing new ones

This is most important when inventing new tags these days because as mentioned above there are already a lot of tags with significant use so that you will hardly be able to invent a new tag without other pre-existing tags being close-by. So how do you proceed here? First of all: Don’t rely too much on the OSM wiki for that. Not all tags with significant use are documented on the wiki and the documentation often does not accurately reflect the actual meaning of tags. Taginfo is usually very helpful to explore what tags exist and how they are used. Using overpass turbo (easiest through the links on taginfo) can be very useful to get more specific insight into how certain tags are used. Be aware that bare use numbers are often misleading due to imports and organized edits sometimes strongly distorting the view and creating the impression that a certain tag is widely used while in fact has only been used by a handful of mappers in an organized or automated fashion. The regional taginfo instances operated by Geofabrik and others can be useful here as well.

Like Jochen already pointed out more than 10 years ago it is not a good idea to invent a new tag that overlaps or re-defines an existing tag with widespread use. This almost always leads to confusion and reduction of data quality of both tags. So what should you do if the idea for a new tag you have in mind overlaps with widely used existing tags?

  • If your tagging idea essentially forms a subclass of an existing tag – like you want to create a tag for vegan restaurants – you should turn that into a subtag for the broader established class. A subtag or secondary tag is a tag that is to be used in combination with another primary tag and develops meaning only in this combination. In this case such a tag already exists of course, it is diet:vegan=yes|only. It can be used in combination with amenity=restaurant but also with other primary tags like amenity=cafe, amenity=fastfood or various types of shops.
  • If your tagging idea overlaps with an existing tag but is also meant to cover things that are not included in said tag you have three options:
    • extend the definition of the existing tag to include what you want to map. This needs to be done with care to avoid devaluing the existing data this tag is used for. If existing data with that tag consistently does not include the kind of thing you want to map and what you want to map is well distinguishable from the existing class of features it might be better to
    • split what you want to map into two separate classes of features – one you map with a subtag to the pre-existing tag and the other with a new primary tag.
    • You can also try to introduce a new tag with the intention to replace existing tagging. This however rarely works if existing tagging is already actively used by a lot of mappers. Most attempts at doing this result in less consistent mapping because they introduce several competing tagging ideas with neither of them universally favored over the others – a development nicely illustrated here. So you need to be very careful with choosing that route. See in particular also the last point below.

Make sure your tag has a positive definition

The success of a new tag also depends on it having a clear positive definition. If you’d for example invent a new tag shop=non_food that lacks a positive definition because it is defined by what it does not sell – namely food. Such tags often turn out to be very broadly and inconsistently used and they tend to discourage the invention of more specific, positively defined tags at the same time. For example someone might tag a museum shop=non_food because it sells tickets to the museum and that is definitely non-food.

Make sure to use a fitting and non-misleading name for the tag

Try to make sure that the key and value you choose for your tag do not have either a more specific or a broader meaning in a native English speaking area. Well known examples where this turned out to be a problem are leisure=park and landuse=forest where the meaning of the tags does not encompass everything that is sometimes referred to in English with the terms park and forest.

This is hard and sometimes poses problems that are not ultimately solvable and you might have to accept an imperfect solution here. But you should try to avoid major problems by evaluating the options for choice of key and value carefully.

Formulate a language independent definition

That might sound like a contradiction because the suggestion to formulate implies use of language and at the same time it should be language independent. How is that meant to work?

The key here is to have a definition that describes the meaning of a tag without referring to culture and language specific terms for that definition. For example you should not define the meaning of natural=beach with a beach. A decent definition for example would be: A landform at the edge of sea or of a lake consisting of unvegetated loose material in grain size between sand and large boulders that is shaped by water waves. Instead of referring to the poorly defined English language term “beach” that many people will associate very different pictures with that definition uses generic terms less open to subjective interpretation (like unvegetated or water waves) to define the use of the tag. While translating the word beach into different languages will result in differences in definition due to semantic differences in the translation of the term translating the more abstract definition will be more accurate and less likely to result in a different interpretation. Being elaborate and possibly even redundant in the definition, delineating the meaning in different terms, can further help with that.

Do not define your tag by referring to Wikipedia

Some mappers try to address the previous point by referring to Wikipedia or copying text from Wikipedia. That is not a good idea. Wikipedia has a completely different perspective on documenting knowledge about the world than OpenStreetMap. Wikipedia documents the commonly accepted meaning of terms based on outside sources. That is (a) bound to change over time and (b) is often self contradicting because different parts of society have different ideas of the meanings of certain terms. We in OpenStreetMap however depend on having a single, globally consistent and locally verifiable definition of tags.

Avoid aggregating unrelated and semantically very different things into a common tag

Tags are meant to semantically structure the geographic reality as we document it in OpenStreetMap. Such a structure is most intuitive if the individual tags are semantically compact or in other words: If in the eyes of the local mapper two features with the same primary tag in the vast majority of cases have more in common than two features with different primary tags. That is not always universally possible. Tagging means classification and classification inevitably requires drawing lines between classes separating things that are semantically similar. A good example are the tags natural=scrub and natural=heath which are distinguished by different heights of scrubs. A tall natural=heath and a not very tall natural=scrub in similar ecological settings might be semantically closer than two very different areas of natural=scrub in different parts of the world. Still natural=scrub is narrow enough in its definition so mappers will intuitively see the semantic similarity of two areas correctly tagged as such no matter how different they are.

I can also formulate this advice in a different way: A newly invented tag should not need to be delineated to too many other tags. Taking again natural=scrub. That needs to and is delineated towards natural=heath and natural=wood regarding the height of the woody plants. It is also delineated towards landuse=orchard for scrubs cultivated for agricultural purposes.

Respect OpenStreetMap’s mapper-centric approach

Tagging in OpenStreetMap is intentionally designed for the ease and clarity from the side of the mapper and not for the usefulness for certain applications of data users. It is therefore not advisable to try designing new tags if you are not using these tags yourself as a mapper or if your main motivation is that you would like mappers to use these tags because of your needs as a data user. Ultimately mapper-centric tagging conventions are in the long term of benefit for both mappers and data users because they ensure high quality data – even if in the short term they sometimes mean additional work for data users to process the data into a form that they need for their application.

Be mindful about geographic diversity

OpenStreetMap in contrast to most other efforts of collecting cartographic data, like for example by public authorities, tries to produce a map of the whole planet. That means dealing with the challenge of an immense geographic diversity around the world. Obviously not every tag is applicable everywhere on the planets so we do and we need to have tags limited to certain settings in application. But it is important to not do so without necessity. Definitions of tags should be practically suitable for all geographic settings. Think for example about the distinction between landuse=farmland and landuse=orchard. That distinction should ideally not be based on a fixed list of products grown (which will inevitably always be incomplete) but on an abstract definition that allows making the distinction for any plant grown on the planet.

Be prepared for failure

This might seem like kind of a pessimistic tune to end this advice with – but it should not be read as such. Everyone – no matter how experienced and knowledgable – should be humble when inventing new tags and be ready to accept that what they suggest might fail. OpenStreetMap’s open tagging system is intentionally designed to deal with failure – with tags being introduced and being used but over time being abandoned in favor of other tagging systems or (more frequently) tags being introduced to replace existing tagging schemes to fix some issues with those but never being broadly adopted because mappers stick to existing tagging. That this happens is not a fault in OpenStreetMap, it is intended.

How tagging in OpenStreetMap looks like is a bit like the geography we map looks like – organically grown, full of flaws, but still largely functional. Yes, you could design a city from scratch on a drawing board to be perfect by some ideal of the perfect city. But it would be really hard to build it and it would be even harder to persuade people to live in such a sterile environment. And even if you solve that problem you would still over time realize that this perfect city is not as robust and adaptable as the organically grown ones. It would be over-optimized to the specific conditions taken as granted by the designer and if those conditions change (for example environmental or cultural factors or demographics) the whole design would be in peril while the organically grown city with all its flaws could organically adjust to changing conditions much better.