July 12, 2021
by chris

State of the Map 2021 – observations

The last weekend the State of the Map conference 2021 took place – the second time as a purely virtual conference, you can find my comments on last year on this blog as well.

This commentary is somewhat preliminary because i did not yet view some of the most interesting talks because of real world obligations and the multiple competing tracks. And since the talks were pre-annouced to be recorded and to be made available afterwards while the panels, workshops and self organized sessions were not i focused on the latter a bit more.

The concept of a virtual conference – second iteration

I presented some general musings on the idea and appeal of a virtual conference in last year’s comments most of which still applies of course. Many of my more radical suggestions to depart from a pure 1:1 virtualization of an in-place conference and embracing the fundamentally new chances of the virtual format have not been followed. I have the impression that many of the influential people in the OSMF and around the conference still view the virtual conference as a temporarily necessary stop-gap replacement of what they actually want – a physical meeting of the international OSM jetset – and therefore don’t want to start things that could in any way be incompatible with moving fully back to what they know and enjoy.

Compared to last year where everything had a fairly improvised feel to it, this year the whole conference framework was much more slick and organized. But to me this is not really a positive thing. It is not that last year things technically did not work on a larger scale and this year i saw plenty of cases of (minor) glitches, in particular when you tried to join a room the browser tab often froze for a considerable amount of time. To me this slickness of this year’s conference was most visible in form of the much tighter constraints, less freedom, more builtin distance between the participants of the conference and less freedom for free form interaction. Specifically:

  • I missed the pads that were last year used for comments, questions and discussions of the talks. The free form and non-linear nature of the pads offered so much more freedom for interaction, working together and expressing yourself than this year’s linear chats and isolated fire-and-forget questions.
  • Linear group chats in general pose a structural disadvantage to slow typers and non-native English speakers and incentivize asocial tactics to dominate a discussion. They promote shallowness in discourse and essentially prevent any more elaborate discussion with multiple participants.
  • Last year’s SotM was essentially an open event where everything (streaming of talks, interactive conference rooms, pads for discussion) was openly accessible to everyone without registration. I saw this as a huge benefit and it contributed a lot to the open get-together flair of that event. This year everything (except for maybe streaming of talks) was only open to registered visitors of the conference (and i am not even sure you could still spontaneously register during the conference).
  • There does not seem to be a permanent record of the chats available to follow up on and continue afterwards.

Don’t get me wrong, this was still overall a well made conference and i saw a lot of effort on side of the organizers to make this suitable and inviting for a large variety of people (including in particular the translation efforts – which i did not check out myself). But from my perspective it was in many ways also a step back compared to last year and i kind of fear that is in a way a (potentially mostly subconscious) step to pave the way back to a primarily non-virtual conference again.

Observing filter bubble maintenance in action

The most memorable impression for me was observing the echo chambers most of the panels at this conference turned out to be and essentially observing filter bubble maintenance in action.

The panels i am talking about here are:

First of all there might be a bit of a cultural difference of course. To me a panel discussion on any serious topic would be an event where people with significant differences in their perspective and opinion on a matter meet, each presenting their views and then discussing them so they come to insights that represent an additional value compared to the sum of its parts. In other words – to me a panel discussion is essentially a dialectic format.

The panels from this conference however were more like the panels of a fan convention where actors and producers of popular performing arts praise themselves and each other for their achievements, tell anecdotes and engage in friendly and shallow banter and the main aim is for everyone to have a good time and enjoy themselves.

Now don’t get me wrong – my issue is not that such events took place at SotM. What i found remarkable however is how the participants of all of these panels clearly thought that they were having a meaningful and insightful conversation while what happened was essentially that they reaffirmed each others’ existing preconceptions. The main result of the events was the reinforcement of the existing filter bubbles of the participants and the parts of the audience on the same wavelength by excluding huge parts of what is already a small and relatively homogeneous subset of the OSM community (the conference visitors) from their perception and celebrating a friendly conversation within that bubble to be a discussion with meaning regarding and representative for the larger reality.

All of these panels had participants that could have provided very valuable contributions to a dialectic or argumentative panel discussion on the subjects discussed (and where i would for example definitely say i could learn a lot from having a conversation with them on the subject in question). But without anyone on these panels questioning the preconceptions and assumptions of the participants and opening their eyes to other perspectives on the matter the intellectual value and the gain in insight of these events for listeners on the subject matter was negligible.

In the Paid-Editing/AI-Mapping Tools panel – which was moderated by a professional scientist who should by profession be used to and experienced in questioning and challenging his views and preconceptions i decided to inquire about this with the question:

Why does your panel consist exclusively of people working for large corporations. Do you think others (small businesses, self employed people, hobbyists) have nothing important to contribute on the matter of paid editing and AI mapping tools or did those you have asked decline to participate?

The answer was sobering: They indicated not to believe that those not represented on the panel in their eyes had nothing important to contribute but they still chose to only invite representatives of large corporations.

Some thoughts on the map reading Q&A

When the call for proposals for this conference was announced i was fairly unsure if to submit something. The most natural thing to submit would have been a talk. But the talks were – like last year – to be pre-recorded to then be streamed to the visitors at the specific times of the schedule only. It seems a bit weird to me – not to say disrespectful to the international audience residing in different time zones – to impose such an artificial constraint (since the talks are prerecorded) and to essentially imitate a live event without it actually being live.

Now having talks pre-recorded and not live makes a lot of sense, in particular it lowers the hurdles to have a talk at such a conference massively. Last year i suggested to make use of the natural possibilities coming with that and to make the talks available to be watched by the visitors whenever is convenient for them in advance and to then asynchronously discuss them and to wrap that up with a live discussion similar to what we had now. But this suggestion was not picked up, probably because it was perceived to be a too radical change and it would be incompatible to moving back to a live non-virtual conference without giving that up again of course.

Anyway – my thought was if the conference organizers want to have the conference as a live event only i would want to do something that actually uses the benefits of live interaction with the audience. At the same time i wanted to try introducing the idea of asynchronous participation (like in my suggestion of making the talks available for viewing in advance) into what i do. That is how i developed the idea of an interactive Q&A workshop.

Here are my spontaneous observations and impression of how it went:

For the pre-event asynchronous involvement of the audience i had created a wiki page and a pad to submit topics to discuss. Speaking live at a virtual event of this form is difficult because there is an almost complete lack of real time feedback from the audience. At the same time it is live so in contrast to a pre-recorded talk you can’t just make cuts and re-record certain parts or the whole thing if necessary. Because i kind of anticipated that my original plan was there to be more audience interaction during the workshop.

But while i had advertised it on the diaries and on the OSM-Carto issue tracker there was only one submission (From Jerry (SK53), many thanks for that) before the start of the conference. And Jerry was not present live in the workshop (which was fine – turning up at the workshop was not a prerequisite for submitting case examples) so i could not start off with an interactive approach to the whole thing. And although during the conference a number of more submissions came in i ended up not having any live dialog in audio/video with the audience. That was a bit unfortunate (and i could probably have tried to encourage that more) – it would likely have been an improvement for both me and the audience to connect a voice and a face to the questions and have more symmetric discussion of the case examples.

In the end most of the interaction took place on the chat and the questions. The splitting into chat and questions in the conference interface was something really confusing and IMO quite pointless. You could only show either the chat or the questions and it is easy to miss additions to one of them while concentrating on the other. I see that there is a potential benefit of having anonymous questions but even that would seem possible while still conflating the two in the same display.

Over the duration of the workshop (which ended up to last for 1:30 instead of just one hour) the interactivity improved (probably mostly because i got more used to keeping an eye on the chat and questions and integrating them into my discussion) but for my taste it was still too asymmetric – as said i would have liked more true dialog.

The other important observation i made: For a live event like that involving a screenshare presentation it would be really good to have a triple-screen-setup. You would essentially need:

  • one screen with the conference interface (video, chat, controls etc.)
  • one screen to share
  • one screen for your notes and staging of materials to share

I only had two screens set up connected (had a third one – but that was on a different computer so of limited use) which made things a bit more complicated.

In terms of the actual content of the workshop – it ended up being a bit less basic map reading and more advanced map design questions – like problems of scale dependent rendering and how to deal with the rural-urban divide. I had prepared some material on the matters of road layering and urban landuse, which was more focused on basic explanation of the map but which i ended up not using because there were more than enough questions from the audience to fill the time. But i am a bit afraid that for some of the less experienced listeners the whole workshop was a bit too advanced towards the end.

What i am kind of wondering is if it would make sense to develop something like a regular recurring event out of this outside SotM. Considering how widely the standard map style is used and referred to in the OpenStreetMap world it is quite definitely severely under-explained and under-discussed on a level above the basic “I want feature x to be rendered” and outside the scope of actual style development. Any input on that idea would be welcome.

Navigating the Maze

May 12, 2021
by chris

Navigating the Maze – part 2

This is the second part of a two part post on roads rendering in OpenStreetMap based digital rule based map styles. In the first part i have explained and analyzed the roads rendering system of OSM-Carto and its issues, here i want to explain and discuss the re-designed road rendering system i drafted for the alternative-colors style and the improvements and new features i implemented in it.

A big warning upfront: This road rendering framework is essentially just a sketch. I will explain in the end what i hope in the long term eventually will come out of it. At the moment this is – as said – a bit sketchy and i don’t want to make any claims that it is overall better by any metric than the existing one in OSM-Carto. Or in other words: It is as much a study on what is possible as much as it is one for what is not. Keep that in mind when you think about actually using this. Also if you work with this technique or in general with the AC-style you should be sure to turn off JIT optimization in the Postgresql configuration. Because that does not deal well with the kind of queries used here apparently.

Rethinking layered rendering

The technical limitations of the OSM-Carto road rendering system i discussed in the first part are to some extent caused by specific design choices in OSM-Carto. But for the most part they are the result of principal limitations of the toolchain used by OSM-Carto. The most fundamental limitation of that is that the order in which you query the data from the database and how this is split into a lot of separate queries is directly tied to the order in which things are drawn on the map. There are some limited options to interfere with that on the raster compositioning level (OSM-Carto is using that a bit already and i use this more in the alternative-colors style – a topic for a different post) but in the end you are pretty fundamentally stuck with this paradigm of query order and drawing order being tied to each other.

Most of the software OSM-Carto is using – specifically Carto and Mapnik – are no more actively developed. That means there is not much of a chance that anything will change about this constraints. This also means that the only major component of the OSM-Carto toolchain that is still actively developed is Postgresql and PostGIS.

This essentially suggests the most promising path for rethinking the roads rendering is to work around the limitations of the layered rendering framework as described by combining all road rendering components into a single layer. If you look at the layer list in the first part with 16 layers you can see that this is a pretty radical step. Fortunately there are already quite a few pre-existing changes in the alternative-colors style that make this a bit simpler. Some layers i have dropped (tourism-boundary), others i had moved outside the road layers range (cliffs), which leaves the following layers to deal with:

  • tunnels
  • landuse-overlay
  • line-barriers
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing
  • highway-area-fill
  • roads-fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom
  • waterway-bridges
  • bridges

In addition i moved the aerialways layer above the bridges (because practically most aerialways will probably be physically above bridges they cross) – although this could in the future be changed and integrated back with taking the layering into account of course. That is still 13 layers of course.

So why would i want to combine all of this in a single layer? This has a number of advantages. Since one layer means one SQL query only you don’t have to query stuff several times from the database and you can avoid duplication of SQL code much better. At the same time addressing some of the fundamental limitations of the OSM-Carto road rendering, in particular the lack of proper layering for highway polygons, becomes much easier and does not require a significant volume of additional code.

And most importantly you don’t have to deal with the limitations of the various methods to define the drawing order i discussed in detail in the first part and the complex and difficult to understand interaction between them because you can simply put the features in the order you want to render them in using ORDER BY in SQL. That means you cannot use group-by and attachments any more of course and for every use of attachments you have to essentially duplicate the features to be drawn in SQL (like using a UNION ALL query). But practically that is relatively simple and adding an ORDER BY statement to the wrapping query of the single roads layer is a much more intuitive and better maintainable way to do this in my opinion.

Layering in SQL

So how does the query combining all these 13 layers and the improvements i implemented look like? Before you go to looking through the more than 1700 lines of SQL code you can have a look at the basic structure where i stripped most of the actual queries, in particular the lengthy column lists which make up most of the query. A more elaborate sketch of the structure is available as well. The query makes fairly extensive use of common table expressions to query different sets of features and then use them in different contexts. On the base level of the query (roads_sublayers) we have the CTEs

  • roads_all – which contains all the linear road features from tunnels/bridges/roads-casing/roads-fill as well as things derived from them (for new features – i will discuss those later),
  • road_areas_all – which contains all the road polygon features from highway-area-casing/highway-area-fill
  • tc_all – which contains the stuff from turning-circle-casing and turning-circle-fill

These CTEs currently are independent of each other. roads_all basically contains the UNION ALL from the existing road layers combining highway lines and railway lines – plus the alternative-colors specific features, which use a number of additional CTEs that reference each other.

Based on the three main CTEs there is a stack of 16 sublayer queries combined with UNION ALL. You can find the non-road layers from the above list in there while the road layers do not match the legacy layers directly. Ignoring for the moment those sublayers connected to new alternative-colors specific features we have:


We don’t need separate sublayers for tunnels and bridges because the sublayers are not what primarily defines the drawing order. The drawing order is defined by the ORDER BY statement that follows the sublayer stack.

That means across all sublayers with a non-null int_bridge attribute (that is everything except the non-roads layers) the features with int_bridge=yes are rendered on top in the order of their layer tag. It does not matter what these are, whether road line bridges, highway polygons, turning circles or waterways. Same for tunnels at the bottom. Apart from that the ordering is similar to the one in OSM-Carto – although i made some adjustments, the effects of which are however mostly very subtle.

Map design in SQL

The most important result of this change is that it allows consistently layering all bridge and tunnel features – no matter if linear roads, highway polygons or waterway bridges – according to the layer attribute as explained in the first part. Here the layering in comparison between OSM-Carto and the alternative-colors style:

Linear roads and waterway bridges layering in OSM-Carto (left) and AC-Style (right)

Highway polygons layering in OSM-Carto (left) and AC-Style (right)

The other fundamental thing i changed and that was also an important prerequisite for the new features i added is to define the line widths for drawing in SQL. This has a two-fold purpose: First it is to make these settings available in SQL for geometry processing and second: To avoid the error prone repetitive way of defining the drawing widths for the different zoom levels as shown in part 1. Parts of this change were already implemented in previous changes where i introduced implicit embankment rendering. This is currently based on a YAML file where you define the different line widths. This file is processed by a python script into alternatively MSS code or an SQL function containing nested CASE statements for selecting the line width based on zoom level and road type. This method is not yet set in stone – an alternative would be to have a separate table in the database containing the line width values. I have now extended approach to also include the casing width values in addition to the overall line width.

Ground unit line width rendering

One thing this change facilitates is allowing to more easily implement ground unit line width rendering at the high zoom levels. The motivation for that is the following: At the low zoom levels roads are universally rendered with a line width larger than their real physical width on the ground. That is universally the case in maps because otherwise the roads would not be visible at smaller scales and even when they would be that would not make for a well readable map. There is however a transit at some point where the drawing width or the roads becomes less than the physical width of the roads. That transit typically happens – depending on latitude, road class and the specific road – at zoom levels between 16 and 18. Drawing the roads more narrow than they are physically is however typically not of benefit for map readability and it also conflicts with the general aim you have when designing large scale maps to move to less geometric abstraction and a depiction closer to the geometric reality on the ground.

Unfortunately you cannot address this simply by increasing the nominal drawing width of the roads in general because not all individual roads of a certain class have the same width of course and what you definitely don’t want at the high zoom levels is to draw the roads wider than their actual width and then obscure other content of the map next to the roads. And fortunately quite a few of the roads in OpenStreetMap have a physical width specified – two million (or about 1 percent) and even more (12 million or 6 percent) have a lanes tag that allows estimating the likely physical width of the road with some accuracy.

To be able to draw lines in ground units on the map you have to be able to convert between map units (also called Mercator meters), ground meters and pixels on the map. The conversion factor between ground units (as tagged with the width tag for example or as estimated from the number of lanes) and map units is called the scale factor of the projection and unfortunately there is no support for directly getting that in PostGIS. I used a generic function that allows approximating that:

  ST_Transform(ST_Translate(geom, 0, 1), 4326),
  ST_Transform(geom, 4326)
)::numeric from (select ST_Centroid(!bbox!) as geom) as p

I do this based on the bounding box of the metatile – not so much for performance reasons but because that ensures geometry processing within a metatile is using the same scale factor – which is essential in some cases. At the zoom levels in question the scale factor will never vary significantly across a metatile. One thing to consider of course in the future is to – for Mercator maps – replace this generic function with the simpler formula specifically for the Mercator projection (because that function is used quite a lot).

The conversion between map units and pixels on the map by the way is the factor !scale_denominator!*0.001*0.28 – so in total you have

pixels = ground_meters/(scale_factor(!bbox!)*!scale_denominator!*0.001*0.28)

This formula is used in carto_highway_line_width_mapped() – which is implemented in roads.sql. Overall the rule for the line width of roads is as follows:

  • If a width is tagged use that as the ground width.
  • If no width is tagged but the number of lanes is tagged then estimate the overall ground width based on the road class and (conservatively estimated) the typical lane width for that type of road.
  • If the ground width determined that way is larger than the nominal line width set by the style than draw it in that width, otherwise in the nominal width.

Currently the style still contains double drawing of the roads in both nominal and ground width (essentially determining the maximum graphically) because i initially had them rendered in different styling. But that should be removed in a future version.

Here is how this looks like at z17 with various different tagged width values – in comparison for different latitudes to show the effect of the variable scale.

Ground unit rendering of roads at z17 at 40 degrees (left) and at 60 degrees latitude (right)

This applies for the roads drawn with the standard casing + fill style. For airport runways the formula is varied to estimate the likely width based on the runway length (using a formula initially suggested for OSM-Carto). For road features with a simple solid/dashed line signature (paths/tracks) the ground width is only visualized at z18 and above – using the styling of footway and track polygons (with some adjustment for tracks to avoid the appearance being too heavy) in addition to the centerline signature. And this ground unit visualization is only used when the drawing width in ground units is large enough to be meaningfully depicted in addition to the centerline.

The ground unit rendering of roads however is not technically as simple as it might seem because of another implicit assumption made by the roads rendering system in OSM-Carto, namely that the drawing order of road classes with the same layer as defined by z_order is in sync with the drawing width progression. That is of course no more universally the case when the tagged width is depicted which would lead to artefacts like the following.

Road junction with artefacts

Solving this requires modifying the road geometries in those cases which is kind of a hassle to do.

Road junction without artefacts

Lanes visualization

Since i already made use of the lanes tagging for estimating the ground unit width of roads i decided to visualize the number of lanes at the high zoom levels as well. As you can see this follows the implicit rules documented which say that trunk roads and motorways are assumed by default to be two lanes. lane_markings=no is indicated with dashed lines instead (click on the image to see).

Lanes visualization at z18+

Lanes visualization at z18+ (click to see also the unmarked version)

In practical use cases ground unit line width rendering and lanes visualization looks like this:

Roads in Strasbourg at z18

Roads in Heidelberg at z18

Roads in Freiburg at z19

This one demonstrating the unfortunate and semantically nonsensical practice of German mappers to tag every road under a bridge as a tunnel.

Of course this style of visualization for both the lanes and the tagged width of the road does not work too well when the number of lanes and the road width changes. This requires mapping and interpretation of additional data, in particular what is mapped with placement=*, which is currently not taken into account.

Turning circles & co.

While we are on the matter of drawing width or road lines – i also extended the existing visualization of turning circles and mini-roundabouts from OSM-Carto with distinct symbolization of both of these as well as turning loops and passing places.

turning circles, turning loops, mini-roundabouts and passing places at z17

turning circles, turning loops, mini-roundabouts and passing places at z17

All of course working in combination with the bridge layering and variable width depiction. The diameter tag for turning circles is also taken into account.

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads ...

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads …

... as well as tracks

… as well as tracks


The next new feature i looked into visualizing is implicitly mapped sidewalks and cycle tracks. Like in case of embankments sidewalks can be mapped both implicitly as an additional tag on the road (sidewalk=both/left/right) as well as explicitly separately as a way with highway=footway and footway=sidewalk. Both are widely used and there is no clear preference among mappers and it is therefore a bit problematic that OSM-Carto only renders the explicit sidewalks. Like in case of embankments the explicit version is easier to render in a primitive fashion while good quality visualization is much easier with the implicit mapping because you can more easily take the relationship between the road and its sidewalk into account.

What i implemented is a demonstration how visualization of implicit sidewalk mapping as well as that of implicit cycleway tracks with cycleway:right=track/cycleway:left=track can work. This is shown at z18 and above

Rendering of sidewalks and cycleway tracks

Rendering of sidewalks and cycleway tracks

and works also in combination with bridges, fords and implicit embankments. They are not rendered on tunnels to avoid obscuring ground level features.

Rendering of sidewalks in combination with various road variants

Rendering of sidewalks in combination with various road variants

And here is how this looks like in real world situations.

Sidewalks in Ladenburg at z18

Sidewalks in Ladenburg at z18

Sidewalks in Portland at z18

Sidewalks in Portland at z18

Steps details

Rendering of roads in their ground unit width also provides more space to visualize additional details on steps. The normal drawing width of steps does not really provide enough room for showing more information without affecting the recognizably of the steps signature. Here is what is shown with nominal drawing width and when the drawing width is large enough at z18+ for wider steps tagged with their width.

Steps details at z18+

Steps details at z18+

Unpaved roads

The distinct depiction of paved and unpaved roads is one of the oldest open issues in OSM-Carto. The difficulty is that it has to work in combination with a lot of other road properties already visualized. Various forms of dashing have been tried and have been found either to be very noisy in appearance or too similar to existing depictions of tunnels (which use a dashed casing), construction roads (which use a dashed fill) or access restrictions (which use a dashed centerline on the fill) to be intuitively understandable without affecting map readability in general. The most promising idea so far has been the use of a fine grained structure pattern for the road fill. That is highly intuitive in its meaning and can be tuned to be fairly subtle. Lukas Sommer has developed this idea some time ago to be ready for use but unfortunately Mapnik did not support 2d fill patterns for line features and while this was added as a feature some time ago this is not yet available in many production environment and especially not in release versions of Kosmtik. And workarounds for this are a bit ugly to implement in the OSM-Carto road rendering system. I have now added this to the AC-style by generating polygon geometries for the unpaved roads using ST_Buffer() and rendering those instead of the normal line features. That is easily possible with the new setup since the line widths are available on SQL level anyway.

The basic styling of the unpaved road rendering used

The basic styling of the unpaved road rendering used – click to see at z18 with and without tagged width visualization

It is important to consider how this looks like in combination with various other road properties – which you can see in the following illustration. There are some issues with the readability of the access restrictions – which is mostly due to their depiction offering very different levels of contrast on the different fill colors. That needs a closer look at and either tuning the access restriction color or revisiting the principal design of this feature in general.

Unpaved road styling in combination with other road attributes at z17

Unpaved road styling in combination with other road attributes at z17 – click to see more

Also here a few examples of practical situations with this styling.

Unpaved roads

Unpaved roads

Unpaved pedestrian area

Unpaved pedestrian area

Open ends

That’s it for the main new road features and improvements i introduced within the new road rendering framework. There a quire a few other tweaks improving consistency in rendering – like getting tunnels and bridges rendered for raceways. And a subtle centerline on runways and taxiways. And as already hinted at there are quite a few open ends in the whole thing design wise that could use further work. But this blog post is already longer than it should be so i will close for now.

Performance of this new roads rendering approach in terms of computing resources required is not bad by the way. While the roads query is often the slowest query of the style – which is not that astonishing because it contains the data previously processed by a lot of different queries – it is not exceedingly slower than other queries. And it is often faster than the water barriers query, which has been the slowest so far and which is why i added an option to disable that for faster rendering.

Technically one important thing left unfinished is that there is currently a lot of duplication of rendering rules between SQL and MSS code. Like for example i have introduced several zoom level selectors in SQL to limit certain queries to specific zoom level ranges (of the form WHERE z(!scale_denominator!) >= 18) which are unnecessarily mirrored by a matching selector in MSS. This is redundant and prone to errors so it is important to come up with some clear principles which parts of the style logic should be in SQL and which in MSS code.

I updated the sample rendering gallery to the style changes by the way and also added quite a few new samples there.

Conclusions on part 2

The main outcome of the whole endeavor of re-designing the road rendering system this way from my perspective is that

  1. yes, there are much better ways to structure rendering of the roads in an OpenStreetMap based map style without taking shortcuts and as a result failing to consistently representing the road topology than the classic OSM-Carto road rendering setup.
  2. The ad hoc implementation of this in the way presented here based on the Carto+Mapnik toolchain is anything but ideal. In particular the difficulty of debugging the SQL code is a major concern.
  3. The map rendering framework that provides a decent platform for implementing the kind of rendering of roads presented here quite clearly still needs to be developed.

My main hope is that what i show here provides valuable additional insights into what exactly the key factors are to be able to develop map design in an efficient, flexible and scalable way (and scalable here is meant from the perspective of style development, not from the side of rendering efficiency – which is an important matter too of course – though pretty much a separate issue). At least to me it seems after working on this project i now have a much better idea of what the core requirements would be for a future map rendering and design system suitable for sophisticated and high quality rule based cartography with zoom level/map scale and styling specific geometry processing.

In terms of the new features implemented it is too early to draw any definitive conclusions i think. Most of them, in particular the variable width ground unit rendering and the lanes depiction i consider to be in a highly experimental state with a lot of open issues – both technically and design wise. The question of what constitutes the core of meaningful OpenStreetMap based cartography for the very high zoom levels (z19+) that goes beyond simple extrapolation of lower zoom level designs plus functioning as a POI symbol graveyard and that does not depend on mappers essentially directly hand drawing the map is still largely undiscussed.

Navigating the Maze

May 11, 2021
by chris
1 Comment

Navigating the Maze – part 1

This post is part of a longer series on map design questions. I have not written much more recently on my work on the alternative-colors style and there is a bit of a backlog of topics that deserve some discussion. I will make a start here with the most heavy of all – the roads. In this first part i am going to explain the OSM-Carto road rendering system and analyze its issues and limitations.

The challenge of depicting roads

The without doubt most complicated part and for any OSM-Carto map style developer the most serious challange when working on OSM-Carto or one of its derivatives is the roads layers. As Paul some time ago compactly put it the roads rendering in OSM-Carto alone is more complex than many other map styles are in total. Roads (and in context of OSM-Carto roads by the way generally refers to all lines and other features along which land navigation of humans takes place – including roads for automobile navigation, paths for navigation by foot, animal or other vehicle as well as railways and traditionally also runways and taxiways on airports, which however have more recently been moved outside of the roads layer system in OSM-Carto) can be complex in their semantics, connectivity and above all in their vertical layering. Representing that in a well readable and accurate way in a map is a challenge to put it mildly.

Most simpler map styles take shortcuts in this field – which tends to show up in map in non-trivial situations. See for example the following case example:

Road example in Dallas, US in OSM-Carto

Road example in Dallas, US in OSM-Carto

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

All maps have their flaws in roads rendering – OSM-Carto in particular with the labeling – which however is a different matter that i am not going to cover here. But beyond that it offers by far the most consistent display of the connectivity and layering of the roads in this intersection.

The way this is done in OSM-Carto i will describe in a moment – but before i want to emphasize that OSM-Carto producing a consistent rendering of how the roads are mapped in OpenStreetMap is of high importance for the quality of data in OSM. It helps ensuring that the information that is necessary for an accurate interpretation of the data is mapped and that the reality is not recorded in a distorting fashion to achieve a certain rendering result. More than in any other field of mapping in OpenStreetMap probably the rules of mapping roads have been developed in connection with how the data is used in rendering while still maintaining the principle that what is mapped and tagged represents verifiable elements of the geography and not just prescriptions how things are supposed to show up in the map. This is an accomplishment that cannot be overstated and that is important to maintain and extend in the future.

The OSM-Carto road layers

To understand the way OSM-Carto renders roads it is important to first understand how the drawing order is defined in the toolchain used by OSM-Carto (Carto and Mapnik). In decreasing priority the drawing order is defined by the following:

  1. The layers of the map style: Data that is rendered in the map is queried from the rendering database in the form of distinct and separate layers and is rendered in exactly the same order. The layer stack is defined in project.mml and starts with the landcover layers (at the very bottom) and ends with labels.
  2. The group-by parameter of the layer in question (in case it is present): The way this works is that the query of the layer is ordered primarily by one column of the results table and by specifying this as group-by parameter you tell carto/mapnik to group the features by that parameter and render them kind of like they were in separate layers.
  3. The attachments in MSS code. That is the ::foo selectors. Those kind of work as a method to accomplish layered drawing within a single layer. But in contrast to the group-by parameter every attachment needs to be coded explicitly, you cannot automatically base attachments creation on some attribute in the data.
  4. The order in which the data is queried in SQL (typically through an ORDER BY statement)
  5. The instances of the drawing rules in MSS code: Those are the prefix with a slash you put in front of the drawing rules. Like a/line-width: 2; b/line-width: 4; Such rules lead to several lines being drawing, one immediately after the other, on a per-feature basis.

OSM-Carto contains exactly two layers that use the group-by feature: The tunnels and the bridges layers for the roads. Overall the roads related layers (and other layers in between) look like this (from bottom to top):

  • tunnels (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill
  • landuse-overlay (military landuse overlay)
  • tourism-boundary (theme park and zoo outlines)
  • barriers (line barriers like fences, hedges etc.)
  • cliffs
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing: with attachment ::casing
  • highway-area-fill
  • roads-fill: with attachments ::halo (low zoom only), ::fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom: with attachment ::halo
  • waterway-bridges
  • bridges (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill

To better understand the terminology used – here an illustration for how the different parts of the road signature are called.

Road signature elements at high zoom levels (z18)

Road signature elements at high zoom levels (z18)

Road signature elements at low zoom levels (z11)

Road signature elements at low zoom levels (z11)

All the individual road line layers (tunnels, roads-casing, roads-fill and bridges) are ordered in SQL primarily by layernotnull (which is the layer tag with an implicit default value of zero assumed) and by z_order – which represents the hierarchy of road classes.

If we ignore most of the non-road layers for the moment and focus on the high zoom levels we have essentially the following drawing order:

  1. tunnels – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)
  2. casings of normal non-tunnel, non-bridge roads – ordered by
    • first turning circles, then polygons, then linear ways
    • layer
    • z_order (+ some other criteria in SQL)
  3. fills of normal non-tunnel, non-bridge roads – ordered by
    • first polygons, then linear ways, then turning circles
    • layer
    • z_order (+ some other criteria in SQL)
  4. waterway bridges – ordered by
    • layer
  5. bridges – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)

This is how this looks like visually:

Layering of linear roads in OSM-Carto

Layering of linear roads in OSM-Carto

That ordering alone however – despite its complexity – does not ensure a consistent depiction of the roads network. That also relies on the specific way the road lines are drawn. To understand this have a look at the following illustration.

Bridge drawing principles with different layers connecting

Bridge drawing principles with different layers connecting

It shows self intersecting road loops as either tunnels or bridges. On the left how this is rendered when mapped as a single way. To actually represent a road crossing over above itself in form of a bridge or tunnel rather than intersecting with itself on the same level you have to split the way into two and assign different layer values.

That the two ways still visually connect where they meet at the top with their ends depends on the way they are drawn with flat line caps for the casing and round line caps for the fill. This leads – as you can see in the rightmost version – to some smaller artefacts when the bridge/tunnel way ends meet at an angle. That brings me to technical mapping rule number one that is good to consider in mapping:

Bridge or tunnel ways connecting at their ends should always have collinear segments where they meet.

The other thing that i hope this explanation shows is the meaning of the layer tag in OpenStreetMap. The wiki is fairly detailed on this but also a bit confusing. What the layer tag does is describing the relative vertical position of the physical object the feature represents, in particular a road, relative to other elements in the immediate vicinity that it is not connected with. It does not define the drawing order in the map (which depends on how exactly the map shows different features), it does not document the vertical position of a feature relative to ground level. This leads to the second technical mapping rule you can take away from this

Use the layer tag sparsely (both in the extent of where you apply it and regarding what cases you apply it in) where it is necessary to avoid ambiguities in the description of the 3d topology of what is represented by the data.

In all other cases there are other tags that will most likely better describe what you want to document – like bridge=*, tunnel=* location=underground, level=*. Or you are trying to do something you should not do (like – as a mapper – drawing the map yourself).

Limitations of this approach

Some limitations of the described system of road drawing are already visible in the samples shown. Others are not and i will have to explain those in the following. Before i do that however i want to mention one additional aspect of the OSM-Carto road rendering system that i have not mentioned so far and that frequently is an additional source of difficulty for developers.

The actual drawing rules for the roads in CartoCSS for large parts serve the different layers together. In particular for example the styling of the roads fill for tunnels, ground level roads and bridges is using the same CartoCSS code. That has the advantage that the different styling variants (see image below) are consistently applied to all the layers. And changes made to that do not need to be duplicated at several places. But it also makes analyzing and understanding the code overall more difficult because after the road features are split into the different layers for rendering them in the proper order these layers are tied together into a meta-maze again for styling.

Different styling variants on all the road classes - many of these can be cross-combined

Different styling variants on all the road classes – many of these can be cross-combined

Beyond that the main principal technical disadvantages of the OSM-Carto road rendering system are:

  • It requires quite a lot of SQL code being duplicated into the different road layers, in particular the tagging interpretation that is being done on SQL level. Some of this has been reduced more recently using YAML node anchors and references – specifically the roads-casing and roads-fill layers as well as the turning-circle-casing and turning-circle-fill layers. That only works when the SQL code is exactly identical though.
  • This of course also means that it requires unnecessarily querying the same data several times from the database. Practically this is not a big issue performance wise but still it is not a very elegant way to do this.
  • Any non-trivial improvements and extensions to be made to the road rendering system typically require additional layers or at least adding to the SQL code that then needs to get synchronized across several layers and adding to the already significant complexity of the whole system.
  • If you look at roads.mss you can find a lot of fairly mechanical repetitions of code to implement the variation of styling parameters across zoom levels. Like the following:

    [zoom >= 13] { line-width: @secondary-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19; }

    Often this is even repeated for other layers or variants – like in this case with:

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @secondary-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @secondary-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @secondary-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @secondary-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @secondary-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @secondary-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @secondary-casing-width-z19; }

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @major-bridge-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @major-bridge-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @major-bridge-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @major-bridge-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @major-bridge-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @major-bridge-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @major-bridge-casing-width-z19; }

    That adds a lot of code volume and is prone to errors. It also makes styling changes that go beyond simple adjustments of the parameters in the CartoCSS symbols used complicated.

On the level of currently present functionality some of the most significant limitations of the current system are (and note these are only those related to the main road layer stack as described – things like road labeling are a different story):

For the polygon features rendered (that is primarily highway=pedestrian and highway=service polygon features) there is no support for bridges and tunnels. That means the above grid example with highway=pedestrian and highway=service polygon features looks like this:

Layering of highway polygons in OSM-Carto (or the lack of it)

Layering of highway polygons in OSM-Carto

Solving that would be possible but you would have to integrate the polygons into all the linear road layers and that would significantly increase their complexity (and the amount of code that needs to be duplicated between them). Same applies for turning circles obviously – although this is a much less significant limitation practically of course.

Another limitation is that while the waterway bridges are correctly layered among themselves they are not interleaved with the road bridges. So a waterway bridge is always drawn below a road bridge independent of the layer tags.

Road layering with waterway bridges in OSM-Carto

Road layering with waterway bridges in OSM-Carto

Practically this is not a big limitation of course (how many waterway bridges are there on Earth where there is a road bridge below?).

What is more important is that for avoiding complexity OSM-Carto has some features not within the road layers that should be included because they can be above or below road features in reality. Specifically that is bus guideways (which have always been separate creating various issues) and airport runways and taxiways (separated more recently) – which is rarely a problem but there are cases where it is (though in most cases of enclosed pedestrian bridges spanning taxiways the bridge is purely mapped as a building and not as a footway).

What i have not covered are the frequent concerns about overall drawing order of roads, in particular road polygons, above other map features, in particular buildings and landcover features. That is a whole other subject and one connected to much more fundamental question about map design and mapping semantics in OpenStreetMap. And it is not something related to the specific design of road layer rendering used by OSM-Carto.

Conclusions on part 1

I tried above to explain how OSM-Carto renders roads and how this system – while being one of the most sophisticated road rendering systems you can find in OSM data based maps – has its limitations. Anyone who tries to implement improvements to map rendering in the style in any way related to roads struggles with those. In the course of my work on the alternative-colors style i have now decided to try re-designing the roads rendering in a fairly fundamental way. How this looks like and what new features and improvements can be achieved with that will be subject of the second part of this post.

April 15, 2021
by chris

On group communication channels

What i want to discuss here a bit is channels and platforms for group communication (that is communication between more than two people) in the context of an open social project like OpenStreetMap. For many years every now and then people turn up on OSM communication channels and express dissatisfaction with either the particular channel or with communication in the OSM community in general – often connected with the suggestion that everyone should move to [fill in your favorite hype of the day platform]. I want to explain here my personal criteria what aspects qualify a communication channel or platform for productive, constructive and inclusive group communication in context of a global multicultural social project like OpenStreetMap.

A few preliminary remarks: First i want to clarify that this analysis concerns group communication in the sense of symmetric communication between more than two participants on equal levels. For one-to-one communication or asymmetric communication (one-to-many with no symmetric back-channel) different criteria apply. Second, a diverse global community like OpenStreetMap depends on there being diverse communication channels used for communication. Concentrating everything on a few channels and platforms would be an issue on its own. OpenStreetMap can only work in a sustainable way if communication and social interaction within the community does not depend on there being a central communication channel everyone engages on. The very idea of OpenStreetMap is that the main database is the element that holds the project together socially and trying to substitute that with some other means would fundamentally weaken the project.

The criteria

  1. Asynchronous communication: In a global, multilingual community like OSM there is inevitably a significant variation in how fluent people are in the language of a specific communication channel. Asynchronous communication (that is communication where there is no expectation, technical need or social pressure for a close timing between a message and reply to that message) can enormously help to level the field in that regard. Not to mention different time zones. Asynchronous communication also helps accommodating participants with intermittent internet access.
  2. Text based communication: For similar reasons text based communication is generally preferably to audio or video communication. Audio or video group communication is of course also almost universally synchronous anyway. In addition text based communication reduces the bandwidth requirements for the participants, makes any kind of filtering massively easier and especially it reduces the participation barrier because it gives participants better control over the content of their communication and the disclosure of private information.
  3. Structured communication: One fairly essential component of group communication with a larger number of participants is transparent visibility of who communicates in reply to whom. The classic way to visualize that is threaded communication. opinions on this differ a lot – some despise the threaded display of communication while others think it is essential for communication in larger groups to work properly. This has a lot to do with digital socialization of people. Those who have become familiar with digital communication without being exposed to the concept of threading often adopt a dislike of this. And with widespread tools and platform (Microsoft, Google) of email communication over the last decade or more by default hiding threading this applies to quite a lot of people. What is fairly clear however is that if a larger number of people participate actively on a communication channel without the communication being structured into threads it will often turn more and more into a synchronous communication channel because asynchronous replies to older communication is perceived to be disruptive by the most active participants of the channel. And this comes with the issues i mentioned above for the asynchronous communication.
  4. Availability of a linkable and immutable permanent public record of past communication: This is one of the most important points. For group communication to be practically meaningful for a larger community it is absolutely essential that you can refer to past communication in conversation in a way anyone can follow and that you can resolve disagreements on the exact nature of past communication transparently. This is easiest through a complete, reliable, immutable, searchable and linkable record of the full history of communication.
  5. Open platforms and communication protocols: For long term sustainability and to avoid dependency on proprietary technology providers. That means in particular the technical interfaces to participate in the communication should be documented and both legally and technically open to be used by everyone without being tied to using specific software.
  6. Practical diversity in communication clients or interfaces: Connected to the previous point – it is of high value if communication protocols are not only open in principle but if there are also different clients and interfaces available that users can choose between according to their individual needs and preferences. Apart from personal preferences and convenience this is also a matter of accessibility – in particular for people with disabilities.
  7. Practically usable flexible and robust ways of filtering communication by the individual user: This is connected to the previous two points. In case of open communication protocols and a variety of communication clients available, one of the most important features those clients tend to distinguish themselves with is filtering options for the communication. And that is of significance on two levels. First it allows people to focus their attention of what they consider important and of interest for them rather than what others consider important. And as a secondary effect it tends to have an immensely civilizing effect on the participants of a channel. If people know everyone else on a channel has the ability to choose ignoring their messages they tend to weigh more carefully what they write and take more into consideration if what they write is likely to be perceived to be of value by others. To put it bluntly: Never underestimate the pacifying effect of a killfile.
  8. Channel is open to everyone and does not require explicit invitation or other access control or discouragement from active participation. Like the required agreement to terms and rules decided on by others than by those who are actually communicating on the channel. This is again essential for diversity in participation. For many people the psychological barrier to actively contribute in a public group communication channel is quite significant so it is important not to create additional hurdles to that. Of course ultimately most, even non-commercial, operators of communication infrastructure ultimately reserve the right to control access to their infrastructure when they consider it necessary but the key here is if that is a possibility that is practically used or not.
  9. Decentralized protocols. This is the ideal solution for the previous issue but practically there are only very few group communication channels that work in a fully decentralized fashion. Historically Usenet was the iconic example of decentralized group communication and since the demise of Usenet there has so far never been any decentralized protocol for group communication that has gained an in any way comparable significance.

And how do different channels compare?

Here is how the various communication channels commonly used in the OSM community fare regarding the criteria mentioned above. I included Usenet/NNTP although his is not practically relevant both in the OSM community and outside of it because is has historically been very important in forming digital group communication culture on the internet and as you can see it also is outstandingly good at meeting my list of criteria. The reasons for the decline of Usenet and NNTP is a whole different story of course (quite an interesting one though – and one that many people have already written on so there is not that much point in touching that here).

The information on Facebook, Telegram and Whatsapp is based on hearsay and therefore not that reliable – i have never used these platforms so i can’t really provide first hand experience.

OpenStreetMap communication channels and platforms

OpenStreetMap communication channels and platforms

Footnotes from the table:

  1. has limited support for structured communication.
  2. The OSM wiki in principle allows structured communication and on talk pages it is established practice to thread messages. But you have to do this fully by hand and it only works if everyone does so diligently. No builtin support for that.
  3. On Usenet server operators used to expire messages so there was practically no permanent public record in the system itself. However everyone running a Usenet server could maintain a record and in the late days of Usenet Google practically served as a reference record.
  4. The forum indicates when a post has been modified after initial posting but no record of the original message is retained. Also admins on the forum can (and do) remove messages from the record (with varying frequency, depending on the channel).
  5. The diaries themselves can be edited without the history being visible but the diary comments cannot.
  6. On the wiki the edit history is openly accessible so it serves as an immutable record (though it can be difficult to use as such practically).
  7. Github has a fairly clever system of allowing edits of communication but still allowing users to transparently see if there have been changes and what has been changed. Github however also allows completely removing messages from the public record in some cases (but it is indicated at least that a message has been removed i think).
  8. While there is no alternative interface to posting changeset comments for reading there is the popular tool by Pascal Neis.
  9. Openness in participation applies to most OSM related mailing lists and forums. There are a few exceptions – specifically osmf-talk is only open for posting to OSMF members. There are thematic and regional lists and forums with culture specific rulesets you need to accept for active participation which are usually developed through consensus of those active on the individual channel (like diversity-talk, regional forums).


The bottom line is: Mailing lists still rule by a huge margin compared to everything else that is used today in the OSM community. But of course the criteria i used are my personal preferences and although i have explained why i see it this way there are others who view what i see as advantages as major disadvantages. And that is fine – as i pointed out in the beginning diversity in communication is essential for a diverse global community like OpenStreetMap. There are some in OpenStreetMap who dislike that and who would want to culturally homogenize the OSM community and would like to nudge everyone to adopt the same communication platforms. That is not going to work of course. If there is one field where you can be absolutely certain that people will always vote with their feet it is group communication channels.


What is interesting is of course to look at what the future might bring. Will there be platforms and channels that can compete with the old school techniques (NNTP and mailing lists) that came out as winners in my analysis regarding the criteria i looked at? At the moment nothing is visible in that direction i think. Focus in development in group communication technology has for quite a few years been in the real time synchronous domain but as i tried to explain above this is of relatively little importance for highly multicultural use cases like we have them in OpenStreetMap. The trend in asynchronous communication has for quite some time been towards monolithic platforms where the client and its user interface are tightly interwoven with the underlying communication protocols (or in other words: Stuff developed around some fancy web interface meant to serve the fashion of the day in UI design). That is unfortunate because it tends to take many years for a highly non-homogeneous community like OpenStreetMap to truly adopt new communication channels en large. And by the time that happens such fashion of the day frameworks have often already reached the end of their short life and are out of fashion and not maintained any more.

March 22, 2021
by chris

Satellite image aquisitions – yearly report 2020

I am a bit late this year but i wanted to continue with my tradition to publish a yearly analysis of the recordings of the two currently operating high resolution open data producing earth observation satellites recording in images in natural color – that is Landsat and Sentinel-2.

Superficially similar analysis is sometimes done by the satellite operators as well of course but in most cases this is not very detailed and in some cases even grossly inaccurate by considering redundancies in published data files as independent recordings (which they obviously are not). Here you can find the most accurate analysis of the spatial distribution of the recordings of Landsat and Sentinel-2 based on the actually published imagery.

Development of total daily published image volume in square kilometers

You can already see from the development of the area covered that for the most part USGS and ESA continued with the same pattern as previous years. The biggest anomaly more recently has been that the USGS apparently for unknown reason stopped off-nadir recordings of Landsat 8 in polar regions for about a year (2019 Northern hemisphere Summer and 2019/2020 southern hemisphere summer) but they have more or less resumed with the previous practice in 2020.

Landsat 8 2020

Sentinel-2 operations continued with the changes made in 2019 (that is extended high latitude coverage in the north at low sun positions with Sentinel-2B, some additional small islands) throughout 2020.

Sentinel-2 2020

More important is what has not changed – and that is in those fields i suggested last year already – so i repeat them here:

For both USGS and ESA:

  • make use of the capacity you have in the northern hemisphere winter to collect more than just sporadic data of the Antarctic.

For the USGS in addition:

  • finally get rid of the last gap in your land area coverage (Iony Island).
  • record off-nadir images of northern Greenland and the central Antarctic on a regular basis (and preferably make use of the flexibility you have here with the chosen path for cloud cover minimization).

And for ESA:

  • stop the non-professional and short sighted catering of special interests and record images based on clear, uniform and transparent criteria.
  • connected to that: rethink your strategy of recording extensive ocean areas around some islands while not recording other larger islands (South Orkney Islands) at all.
  • record with Sentinel-2A and Sentinel-2B the same way or explain clearly why you don’t.

And here as in past years all of the maps showing image coverage together:

year day night day pixel coverage
2014 LS8, LS7 LS8 LS8
2015 LS8, LS7 LS8 LS8
2016 LS8, LS7 LS8 LS8, S2A
2017 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2018 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2019 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)
2020 LS8, LS7 LS8 LS8, S2A, S2B, S2 (both)

March 16, 2021
by chris
1 Comment

How to invent new tags in 2021

Back in 2010 Jochen wrote a practical guide How to invent new tags providing some useful advise to mappers what to consider when invention new tags. Back then however the world of OpenStreetMap was very different. There were far fewer established tags than today and OpenStreetMap was still very young and most people the guidance was written for were new to OSM and to the whole concept of cartographic data collection.

Today in 2021 OpenStreetMap and the typical situation of people inventing a new tag looks very different. There are so many widely used tags that you can hardly any more invent a tag on the green field so to speak without taking into consideration already existing tags in the semantic vicinity of the tag you consider inventing. And most people who get into the position of inventing a new tag have either already been involved in OSM for quite some time and have developed a firm opinion on how things should be mapped and tagged (not always matching how things are actually mapped and tagged) or come from outside of OpenStreetMap and have a firm opinion on how cartographic data collection should look like shaped outside of OpenStreetMap.

Therefore i decided to formulate a new guidance focusing on things that i consider practically relevant when inventing new tags nowadays. This is mostly based on years of observing and studying tags and tag use as an OSM-Carto maintainer and my observations what decides on if a tag becomes a meaningful tag being adopted and consistently used by mappers and also on observing countless tagging proposals over the years. It is not meant to replace Jochen’s guidance which focuses a lot on formal and technical aspects which i will not cover here but to supplement it with updated guidance on semantic, social, cultural and geographic considerations.

Design tags to be practically verifiable

Verifiability is the core of mapping and tagging in OpenStreetMap. While we have a number of poorly verifiable legacy tags in OpenStreetMap (like tracktype) it is a fairly universal rule that newly invented tags that are not practically verifiable fail to become successful tags. They either fail to become adopted by mappers at all (like view) or worse: they fail to develop a well defined meaning in practical use despite being widely used and therefore lead to mappers wasting time to tag information that is practically useless because it is not consistent.

Practical verifiability means that a mapper needs to be realistically able to determine locally on the ground and independent of other mappers or external data sources if a tag is correct or not. In some cases tagging ideas are inherently subjective (like the usability of a road with a certain class of vehicle). Sometimes non-verifiable tags however also result from being invented as a subjective aggregate of several individually verifiable aspects which could individually be tagged in a verifiable fashion (like to some extent natural=desert).

Another distinct class of non-verifiable tags is data that is defined not through something locally observable but through authoritative information from outside of OSM. This for example applies to ratings or restaurants or other establishments. Such data simply does not belong in OpenStreetMap.

One important aspect of verifiability is the practical ability to localize features (and to delineate them when mapping with linear ways or polygons) in a verifiable fashion. Something might verifiably exist but could still not be practically mappable in OpenStreetMap because it is not practically localizable.

Do not invent tags for storing redundant or computable information

Sometimes tags are invented where mappers are supposed to tag information derived from other data in OSM or outside of OSM or that duplicates existing data in an aggregated fashion. Examples are the tagging of the length of linear features or of the prominence of peaks.

Such tags – as convenient as they might seem to data users – are a bad idea because they take valuable time of mappers to compute and aggregate information that could also be automatically computed and there is nothing to ensure that the information is kept up-to-date so it will never be reliable so data users who want to have this kind of information in good quality will always try to compute it themselves.

Be mindful of existing tags when designing new ones

This is most important when inventing new tags these days because as mentioned above there are already a lot of tags with significant use so that you will hardly be able to invent a new tag without other pre-existing tags being close-by. So how do you proceed here? First of all: Don’t rely too much on the OSM wiki for that. Not all tags with significant use are documented on the wiki and the documentation often does not accurately reflect the actual meaning of tags. Taginfo is usually very helpful to explore what tags exist and how they are used. Using overpass turbo (easiest through the links on taginfo) can be very useful to get more specific insight into how certain tags are used. Be aware that bare use numbers are often misleading due to imports and organized edits sometimes strongly distorting the view and creating the impression that a certain tag is widely used while in fact has only been used by a handful of mappers in an organized or automated fashion. The regional taginfo instances operated by Geofabrik and others can be useful here as well.

Like Jochen already pointed out more than 10 years ago it is not a good idea to invent a new tag that overlaps or re-defines an existing tag with widespread use. This almost always leads to confusion and reduction of data quality of both tags. So what should you do if the idea for a new tag you have in mind overlaps with widely used existing tags?

  • If your tagging idea essentially forms a subclass of an existing tag – like you want to create a tag for vegan restaurants – you should turn that into a subtag for the broader established class. A subtag or secondary tag is a tag that is to be used in combination with another primary tag and develops meaning only in this combination. In this case such a tag already exists of course, it is diet:vegan=yes|only. It can be used in combination with amenity=restaurant but also with other primary tags like amenity=cafe, amenity=fastfood or various types of shops.
  • If your tagging idea overlaps with an existing tag but is also meant to cover things that are not included in said tag you have three options:
    • extend the definition of the existing tag to include what you want to map. This needs to be done with care to avoid devaluing the existing data this tag is used for. If existing data with that tag consistently does not include the kind of thing you want to map and what you want to map is well distinguishable from the existing class of features it might be better to
    • split what you want to map into two separate classes of features – one you map with a subtag to the pre-existing tag and the other with a new primary tag.
    • You can also try to introduce a new tag with the intention to replace existing tagging. This however rarely works if existing tagging is already actively used by a lot of mappers. Most attempts at doing this result in less consistent mapping because they introduce several competing tagging ideas with neither of them universally favored over the others – a development nicely illustrated here. So you need to be very careful with choosing that route. See in particular also the last point below.

Make sure your tag has a positive definition

The success of a new tag also depends on it having a clear positive definition. If you’d for example invent a new tag shop=non_food that lacks a positive definition because it is defined by what it does not sell – namely food. Such tags often turn out to be very broadly and inconsistently used and they tend to discourage the invention of more specific, positively defined tags at the same time. For example someone might tag a museum shop=non_food because it sells tickets to the museum and that is definitely non-food.

Make sure to use a fitting and non-misleading name for the tag

Try to make sure that the key and value you choose for your tag do not have either a more specific or a broader meaning in a native English speaking area. Well known examples where this turned out to be a problem are leisure=park and landuse=forest where the meaning of the tags does not encompass everything that is sometimes referred to in English with the terms park and forest.

This is hard and sometimes poses problems that are not ultimately solvable and you might have to accept an imperfect solution here. But you should try to avoid major problems by evaluating the options for choice of key and value carefully.

Formulate a language independent definition

That might sound like a contradiction because the suggestion to formulate implies use of language and at the same time it should be language independent. How is that meant to work?

The key here is to have a definition that describes the meaning of a tag without referring to culture and language specific terms for that definition. For example you should not define the meaning of natural=beach with a beach. A decent definition for example would be: A landform at the edge of sea or of a lake consisting of unvegetated loose material in grain size between sand and large boulders that is shaped by water waves. Instead of referring to the poorly defined English language term “beach” that many people will associate very different pictures with that definition uses generic terms less open to subjective interpretation (like unvegetated or water waves) to define the use of the tag. While translating the word beach into different languages will result in differences in definition due to semantic differences in the translation of the term translating the more abstract definition will be more accurate and less likely to result in a different interpretation. Being elaborate and possibly even redundant in the definition, delineating the meaning in different terms, can further help with that.

Do not define your tag by referring to Wikipedia

Some mappers try to address the previous point by referring to Wikipedia or copying text from Wikipedia. That is not a good idea. Wikipedia has a completely different perspective on documenting knowledge about the world than OpenStreetMap. Wikipedia documents the commonly accepted meaning of terms based on outside sources. That is (a) bound to change over time and (b) is often self contradicting because different parts of society have different ideas of the meanings of certain terms. We in OpenStreetMap however depend on having a single, globally consistent and locally verifiable definition of tags.

Avoid aggregating unrelated and semantically very different things into a common tag

Tags are meant to semantically structure the geographic reality as we document it in OpenStreetMap. Such a structure is most intuitive if the individual tags are semantically compact or in other words: If in the eyes of the local mapper two features with the same primary tag in the vast majority of cases have more in common than two features with different primary tags. That is not always universally possible. Tagging means classification and classification inevitably requires drawing lines between classes separating things that are semantically similar. A good example are the tags natural=scrub and natural=heath which are distinguished by different heights of scrubs. A tall natural=heath and a not very tall natural=scrub in similar ecological settings might be semantically closer than two very different areas of natural=scrub in different parts of the world. Still natural=scrub is narrow enough in its definition so mappers will intuitively see the semantic similarity of two areas correctly tagged as such no matter how different they are.

I can also formulate this advice in a different way: A newly invented tag should not need to be delineated to too many other tags. Taking again natural=scrub. That needs to and is delineated towards natural=heath and natural=wood regarding the height of the woody plants. It is also delineated towards landuse=orchard for scrubs cultivated for agricultural purposes.

Respect OpenStreetMap’s mapper-centric approach

Tagging in OpenStreetMap is intentionally designed for the ease and clarity from the side of the mapper and not for the usefulness for certain applications of data users. It is therefore not advisable to try designing new tags if you are not using these tags yourself as a mapper or if your main motivation is that you would like mappers to use these tags because of your needs as a data user. Ultimately mapper-centric tagging conventions are in the long term of benefit for both mappers and data users because they ensure high quality data – even if in the short term they sometimes mean additional work for data users to process the data into a form that they need for their application.

Be mindful about geographic diversity

OpenStreetMap in contrast to most other efforts of collecting cartographic data, like for example by public authorities, tries to produce a map of the whole planet. That means dealing with the challenge of an immense geographic diversity around the world. Obviously not every tag is applicable everywhere on the planets so we do and we need to have tags limited to certain settings in application. But it is important to not do so without necessity. Definitions of tags should be practically suitable for all geographic settings. Think for example about the distinction between landuse=farmland and landuse=orchard. That distinction should ideally not be based on a fixed list of products grown (which will inevitably always be incomplete) but on an abstract definition that allows making the distinction for any plant grown on the planet.

Be prepared for failure

This might seem like kind of a pessimistic tune to end this advice with – but it should not be read as such. Everyone – no matter how experienced and knowledgable – should be humble when inventing new tags and be ready to accept that what they suggest might fail. OpenStreetMap’s open tagging system is intentionally designed to deal with failure – with tags being introduced and being used but over time being abandoned in favor of other tagging systems or (more frequently) tags being introduced to replace existing tagging schemes to fix some issues with those but never being broadly adopted because mappers stick to existing tagging. That this happens is not a fault in OpenStreetMap, it is intended.

How tagging in OpenStreetMap looks like is a bit like the geography we map looks like – organically grown, full of flaws, but still largely functional. Yes, you could design a city from scratch on a drawing board to be perfect by some ideal of the perfect city. But it would be really hard to build it and it would be even harder to persuade people to live in such a sterile environment. And even if you solve that problem you would still over time realize that this perfect city is not as robust and adaptable as the organically grown ones. It would be over-optimized to the specific conditions taken as granted by the designer and if those conditions change (for example environmental or cultural factors or demographics) the whole design would be in peril while the organically grown city with all its flaws could organically adjust to changing conditions much better.

January 28, 2021
by chris

Snow in Europe

Compared to the previous years this winter seems to turn out to be relatively snow rich in many parts of Europe. On this occasion here a few satellite image impressions of that.

Snow in England on January 25

First a view of snow in the lower parts of much of England – which is not really rare but still not a very common impression either.

Snow in Spain on January 12

Then what is really unusual is the recent extensive snowfall and snowcover over large parts of central and northern Spain including the City of Madrid as shown in the second picture here.

Snow in Greece on January 20

Much more common than in Spain snowfall is in the eastern Mediterranen. I have showed a winter view of the island of Crete before.

Above view shows a recent impression of Greece and the Aegean Sea – featuring snow in many mountain areas including many of the islands, in some cases reaching down to sea level like on the island of Limnos – here a more detailed view of that.

Limnos with snow on January 20

First three images are based on Sentinel-3 data, the last one is created from Sentinel-2 data.

December 16, 2020
by chris

Iceberg A-68 meets South Georgia

The large iceberg that has broken off the Antarctic peninsula in 2017 and that has slowly drifted across the Drake Passage during the past year – i showed some pictures from the beginning of that journey already – has now reached the island of South Georgia which results in impressive pictures demonstrating the enormous size of the iceberg. Like this view from Sentinel-3B from December 14:

A-68 meets South Georgia in December 2020

Here the locator map:

The Iceberg might become grounded in shallow water near the island and could this way significantly affect the ecosystem in the area. Note however that while an iceberg of this size is fairly rare, smaller yet large icebergs are very common around South Georgia because currents and winds in the area lead to a large number of icebergs from the Weddell sea area to drift north along a similar route as A-68. So with a long term historical view such a visit of a very large iceberg of more than a hundred kilometers in size near South Georgia is most certainly not unprecedented, even though previous similar events were not recorded in as much detail by earth observation satellites of course.

November 14, 2020
by chris

The OSMF – changes during the past year and what they mean for the coming years – part 2

In the first part of this blog post i summarized the most important developments in the OpenStreetMap Foundation (OSMF) during the past year from my perspective. In this second part i will – based on the observations from the past year and recent trends being visible – give a lookout on how the OSMF could develop in the coming years.

After my previous piece on the OSMF and the perspective for the upcoming general meeting, Severin Menard has published an interesting and likewise critical take on the current situation of the OSMF. I agree with most of what he wrote and his culturally somewhat different perspective on things is very valuable. Definitely recommended for everyone to read.

Corporate takeover – it has already happened

There is one thing however i disagree on with him – the assessment of a corporate takeover as a risk of the future. During the past weeks there have also been some half baked initiatives started (last minute before the general meeting as usual) from within the OSMF board towards resolutions meant to protect the OSMF against external takeover. My view is that these measures and the focus on a defensive strategy against an external attack aimed at controlling the OSMF are meanwhile setting the wrong priorities because the corporate takeover has essentially already happened behind the scenes without that being clearly visible to the outside.

Large corporate OSM data users are not really that interested in staging a coup in the OSMF and run the OSMF themselves at this time. That would be expensive to do and bear a large spectrum of fairly big risks and also it would – if successfully executed – immediately lead to a struggle for control between the major corporate players. The main goal of corporate actors with the OSMF is and has been for some time to prevent meaningful regulation of corporate activity in OSM and of OSM data use based on the OdbL. If that goal is secured, a secondary goal would be for the OSMF to serve as a shared neutral intermediary platform between the different corporations to steer independent volunteer activity in the OSM community in the corporate data users’ collective interests.

The parts of the OSMF board not affiliated with corporate or organized interests seem to have, during the past years, developed the idea that these corporate interests can be negotiated with and that the future of OpenStreetMap lies in compromising between the organized and corporate interests and the core ideas and values of the project. This quite clearly is an illusion – expecting a big corporation like Facebook to make compromises with an insignificant player like the OSMF is at best naive.

If the goal to prevent meaningful regulation of corporate activity through the OSMF has been accomplished permanently, corporations have no reason to oppose effective takeover prevention of the OSMF – because that would help protecting their position in the OSMF against third parties gaining influence. And effective prevention of meaningful regulation does not even depend on there being a pro-corporate majority among the OSMF members because the corporations have other significant channels of influence in the OSMF now (through their financial constributions, through their participation in the working groups and through lobbying of the board on non-public channels).

We will in the near future have a fairly good test case for how robust the ability of corporate interests in the OSMF is in preventing meaningful regulation in form of the attribution guideline that has been worked on during the past years. There are essentially three scenarios of what could happen:

  • The OSMF board decides to adopt a guideline roughly based on the corporate wishlist from the LWG with some minor adjustments for the optics, to maintain the impression that it is not directly what corporate lobbyists have written. That seems the most likely scenario at the moment but it bears the strong risk that the craft mapper community will openly oppose this interpretation of the ODbL which would fundamentally endanger the OSMF’s position in the OSM community.
  • The OSMF board adopts a guideline reflecting the community consensus reading of the ODbL (likely similar to what i drafted) that unconditionally requires attribution that practically makes the user aware of the origin of the data. That would be strongly against the corporate interests and corporations would certainly do everything within their power to prevent that (including withdrawing funds which would leave the OSMF in financial peril because the strongly increased costs make it depend on regular corporate contributions).
  • A decision on the matter is avoided by dragging out the process indefinitely. Although not ideal, because it would be a kind of unstable situation, this would be acceptable for the corporations because it would maintain the status quo of the OSMF not becoming active against data users with insufficient attribution. It would also avoid an open break with the OSM community, although there would be likely increasing pressure from the OSM community on the OSMF to get active in cases of insufficient attribution by the OSMF’s corporate financial contributors.

Some will likely reject my idea that the corporate takeover of the OSMF has already happened. They will argue that if that was true, corporations would push much more aggressively for their interests. I don’t think that is the case though. As explained, the primary interest large corporations have in the OSMF is not positively accomplishing something, it is preventing things negative for their interests from happening. Accomplishing that is much more valuable for them than anything they could proactively try to push for in the OSMF.

What we will certainly see in the coming years is corporations trying to consolidate their influence on the OSMF, in particular by more and more corporate employees being encouraged to volunteer and being paid for work on the OSMF in the working groups, on the board, in committees and other ways. This will happen rather quickly in most of the working groups probably, supported by the move of the OSMF to position itself more like a corporate actor which, as i have explained before, is likely going to have a negative effect on motivating volunteers without career interests in OSM for the OSMF. My estimation is that in 1-2 years a solid majority of the people engaged actively in the OSMF in one way or the other is either employed in some form in an OSM related job or otherwise has a carreer interest at least partly motivating their involvement in the OSMF. Most of them will be employed by corporate OSM data users or organizations around OSM like HOT. Currently in the OSMF board for example we have already at least three members to whom this applies, two corporate employees (Mikel and Paul), one small business employee (Rory).

Centralization of the OSMF

The other big upcoming trend in the OSMF i can observe is an increased centralization of power towards the board. The OSM community overall is highly decentralized and the OSMF has always been kind of an abnormality within that with its hierarchical structure. But traditionally in the OSMF most of the work has been done in the working groups – including development of policies – and the working groups had a high degree of independence, starting from being created by grassroot initiative from within the community, to allowing also non-OSMF members to participate and to by convention allowing board members to contribute, but not allowing them to have a lead role. We more recently, however, see a trend towards more and more policy being either actively influenced by the board or being developed by the board itself from the start. Early manifestations of this trend were the already mentioned Crimea decision (where the board overruled standing policy developed by the working groups) and the organized editing policy (where the board flat out rejected the first draft of the DWG and demanded a more lenient policy). This year we saw a large number of cases where the board created internal policy, often presenting the results as a done deal without having an open discussion – like in case of the diversity statement – further emphasizing this trend.

In addition, as i have discussed in the first part, we saw during the past year the establishment of several committees (a concept that previously did not exist in the OSMF) – put together by and under direct control of the board. And for this year’s general meeting we have an AoA change proposed that allows the establishment of committees consisting of board members and OSMF members and to delegate any powers of the OSMF board to these. In other words: It would allow the OSMF board to recruit volunteers or paid staff (paid by either the OSMF or by third parties) and delegate any kind of function of the board to them. Obviously, such committees would be under immediate and absolute control of the board, people could become members of such committees only by appointment by the board and the board could dissolve or remove powers from such a committee at any time. But, in contrast to the working groups – which don’t have any formal powers and depend for any meaningful decisions on approval by the board – the commitees could be equipped with any of the formal powers and rights the board has within the OSMF.

The effect the establishment of such committees would have is a massive shift in power within the OSMF from the working groups towards the board. Currently the board is essentially limited in what they can do by their numbers. Board members are not able to delegate their formal powers to others. Work requiring more hands can only be done by the working groups, which have a high degree of independence. If the mentioned resolution passes, the board would essentially no more depend in any way on the working groups of the OSMF – they could assign any tasks so far in the remit of the working groups to the committees they appoint and control without any restrictions.

We can already see right now that the board is increasingly trying to take direct influence on what the working groups do and how they work. In public communication they have expressed an interest in reshaping the remit of the CWG and to reactivate the EWG (which – since the EWG is completely inactive right now outside Google summer of code management – amounts to nothing less than bootstrapping a new working group, something the current board at the beginning of the current term still considered inappropriate).

An interesting test case for the tendency of the board to seek direct control over what happens in the OSMF is the establishment of the planned software dispute resolution panel. The DWG has expressed interest in this task but it seems likely that the board will prefer to directly control the composition of this panel.

What is not visible is any indication that this trend in centralization of power in the OSMF is balanced in any way by more independent oversight over decisions and processes within the OSMF. The idea, for example, that the local chapters could gain some kind of meaningful power in the OSMF’s structure and processes, or that the OSMF could share control over key elements of the OSM infrastructure (database rights and the contributor database) with the local chapters, is not in sight. Given how keen the current board seems to be to gain more immediate control over everything within the OSMF, it seems unlikely that the board will be willing to relinquish any power voluntarily any time soon.

Decreasing diversity and brain drain

Another trend that has already been visible for quite a few years is the increasing difficulty for the OSMF to attract competent and committed volunteers. As i have pointed out in context of the plans of the OSMF of paying people for work, this will likely have a significant negative effect on volunteer motivation.

Now, if you see this problem only as a numbers game, it is not unlikely that it is possible to fill this deficit with paid people (paid either by the OSMF or by external interests) or by people which view volunteering in the OSMF as a career builder. As indicated above, this development is already in progress. There are a number of effects this is going to have:

  • Parts of the OSMF (working groups, committees) will become increasingly dominated by culturally narrow circles of people with shared interests (like their careers or shared interests of outside organizations they are affiliated with) – interests which are distinct from the funadamental goals and values of OpenStreetMap. This process is going to be self emphasizing because once this happens the working group or committee in question becomes increasingly unattractive to anyone outside this narrow spectrum who does not share the interests of the group – even if that group of people considers themselves to be open and welcoming to others.
  • It will probably be in particular the deeply committed OSM community members who have many years of experience in the project for whom volunteering in the OSMF will become increasingly unattractive and who will likely seek to contribute in functions outside the OSMF. That means there is likely to be a quite significant brain drain of competency and experience.
  • The de facto goals of the OSMF will increasingly drift away from the goals and values of the OpenStreetMap project to the special interests and culture specific values of those who happen to be dominating the organization. From within the OSMF and its communicative echo chamber it will potentially not be so visible – the prevailing opinion on the OSMF board often already seems to be that that OSMF’s and OpenStreetMap’s goals are neccesarily identical, in an L’état, c’est moi kind of way.

Seeking influence on OpenStreetMap

As a result of what i wrote in the previous sections it seems likely, and there are already indications for it, that the OSMF is going to increasingly try to take influence on the OpenStreetMap project itself, getting in conflict with the basic premise of the OSMF of supporting the project but not controlling it. The fields in which this is likely going to happen are in particular:

  • Community communication channels: One prevailing narrative from within the OSMF more recently has been that “fragmentation of communication” in the OSM community is a big problem that needs to be addressed. That terminology is in itself interesting by the way – the same thing, when considered positively, is called diversity, when deemed negatively it is framed fragmentation. But that just as a side note. We can expect there to be strong pushes from within the OSMF to try taking tighter control over communication channels hosted by the OSMF by imposing behavior regulation. If these attempts will be successful will need to be seen. What can be said with near certainty is that if that happens it would have the opposite effect of what it claims to want, it would more strongly fragment the communication within the OSM community by essentially squeezing out those who do not want to subject themselves up-front to a narrow culture specific communication rule set in their communication and who would move to communication channels outside the control of the OSMF.
  • The OpenStreetMap website: Traditionally the OSMF has no say in the design of the OpenStreetMap website – it is developed in the classic OSM tradition of do-ocracy and consensus. But there have been increasing voices from within the OSMF expressing the desire to less prominently feature the map as the symbol and the main instrument of inter-cultural cooperation in the OSM community and re-design it more like a corporate website and more proactively communicating the OSMFs interests to the visitor. We will likely see a move in that direction in the coming year – if that is going to be successful is not sure in my eyes though.
  • Mapping and tagging: The OSMF board has last year already made a move that could lead towards taking an influence on mapping and tagging in OpenStreetmap, something that is in principle strinctly forbidden by the mission statement of the OSMF, through the plans for a software dispute resolution panel. The primary purpose of the panel is resolving conflicts w.r.t. development decisions of the iD editor which are primarily conflicts about tagging presets and validation rules. Deciding on such conflicts (which is what such a panel would need to do) would inevitably amount to making tagging decisions. The OSMF of course has no means to enforce such decisions currently beyond their influence on the iD development, but still, the potential of attempts in that direction is definitely there. As i have explained above, a secondary interest the financiers of the OSMF have is steering the craft mapper community into a direction beneficial for their data uses. Members of the current OSMF board have already expressed that they deem usefulness of the geodata collected in OSM as the primary goal of the project – in contrast with the traditional goals and values of the project which put in foreground community cohesion. And if you assume usefulness of the data to mean usefulness for the economically important corporate data users’, interests of the corporations and goals of the OSMF board seem to align here. Still, in contrast to the previous two points where clear trends and already visible, on this matter i would not make a clear prediction.


Now this outlook onto the next years is obviously rather bleak and, indeed, my view of the near future of the OSMF is fairly dark. For OpenStreetMap in general i see this, however, as a valuable chance to emancipate itself from the OSMF a bit more – which, no matter where the OSMF is steering, would be a healthy thing. And i also want to point out a few chances i see for the OSMF in the near future. As i will explain, the likeliness of these actually happening is quite small – yet i find it important to show that the OSMF is not inevitably doomed, but that what happens still depends on the people in positions of power making responsible decisions and there is still a possibility for developing into a much more positive direction. Two examples:

  • With the Brexit and the possibility that a move of the OSMF from the UK to the EU is advisable, there is a real chance in the context of such a move to restructure the OSMF from the highly hierarchical and centralized form essentially required by British company law, into a federated organizational structure with checks and balances and meaningful subsidarity rules, as it would be appropriate for an organization within the highly decentralized OSM community. This is, unfortunately, highly unlikely to happen even if a move of the OSMF takes place, because it would essentially require the OSMF board to decide to give away much of its power to more federated authorities in a new organizational structure. But the possibility is there – there are no meaningful principal hurdles against implementing such a change beyond the general difficulty of moving the organization in the first place.
  • The OSMF has the chance, due to the strongly increased public visibility in the last years, to gather the necessary means (and i mean this independent of contibutions of larger corporate financiers with either explicit or implicit strings attached) to return to and to concentrate on positioning itself as a neutral infrastructure provider for the OSM community. With that i mean to – like a state providing roads and other infrastructure to its citizens – provide neutral infrastructure for everyone to use without discrimination. This idea is not the same as desiring a very small OSMF. Infrastructure can be extensive and expensive. If the OSMF would want to offer each local community around the world the infrastructure to design and run their own real time updated map for example, the costs of this would be fairly massive but at the same time it could be highly beneficial for supporting cultural diversity in the OSM community. The key here would be neutrality and non-discrimination. Much of what the OSMF currently does in new money spending is not – the already cited idea to financially support people whose work we know and enjoy is fundamentally incompatible with this idea. Unfortunately, like in the previous point, this vision would require restraint from the OSMF board to resist any urge to govern and actively steer OpernStreetMap in a direction they deem desirable – something i do not see being likely to happen with the current board.

As i wrote in part 1 of this post, i present my prediction for the direction in which the OSMF is headed here to be proven either right or wrong by what will actually happen. And i would be happy to be proven wrong – either by what the future will actually bring or beforehand by convincing arguments brought up by those who see a different development coming. In other words: I openly challenge anyone who disagrees with my analysis to present and argue for their own predictions in public.

One thing the OSMF board seems to have lost over the past year is the willingness to expose their vision and their plans and ideas to an open discussion and try to convince the OSM community of the merits of their ideas in open discourse. Probably at least in parts because with the money the OSMF has, it seems easier to just buy the implementation of your plans instead of engaging with and convincing the community to support you.

And i call to everyone in the OSM community to not accept this. Whether you have the feeling the OSMF is going in the right direction or not, whether you agree with my analysis here or disagree: You should require people in the OSMF to present arguments and reasoning that convince you of the merit and the solidity of their plans and their actions and require them to present a meaningful vision and expectations for the success and outcome of their actions which can be tested against the actual future development just like my predictions. What is best for OpenStreetMap is not the median of all interests articulated, weighed with the amount of money behind them. What is best for the project can only be decided through arguments and reason.

November 13, 2020
by chris

The OSMF – changes during the past year and what they mean for the coming years – part 1

In my previous blog post regarding the OpenStreetMap Foundation (OSMF) i indicated that i was going to write a bit on the high impact changes that have happened in and around the OSMF during the past year and what implications they are going to have for the coming years – both on the OSMF internally as well as the relationship with the larger OpenStreetMap community.

Some of the changes i list in the following are probably universally agreed on to be of significant impact on the future of the OSMF and the OpenStreetMap project, while others are probably changes others consider insignificant procedural modifications – some of them are possibly even changes that the OSMF board and working groups have not been consciously aware of that they happened. You can obviously have different opinions on this kind of thing – significance of changes is evidently a subjective assessment. The reason i list them here is because they play a role in the predictions on the future development of the OSMF i will discuss afterwards.

Communication habits and style

One of the most impactful changes during the past year – yet also probably one of the more subtle ones, especially for those not that familiar with the OSMF organizational culture during the previous years – is the change in communication habits and style within the OSMF, in particular regarding the board. I already mentioned that the past year was within the OSMF characterized by a fairly massive rollback in transparency and a decrease in open and public deliberation on decisions. Identifiable components within that field are in particular

  • The OSMF board has closed half of their meetings to the OSMF members. The closed meetings are not officially called board meetings and they are created in addition to the monthly official meetings allowing the board to pretend that transparency has not decreased but for everyone who has listened in on board meetings in previous years and now can see that the character of the public meetings has changed fundamentally: there is almost no deliberation on decisions any more, most decisions are undisputed – meaning that the decision has often essentially already been made in non-public communication and the vote in the public meeting is just pro forma.
  • The volume of non-public internal communication of the OSMF board has increased massively – probably at least by an order of magnitude – compared to previous boards. This is not publicly directly visible but can be derived from public statements that have been made and is confirmed by non-public statements of individual board members. At the same time, the volume of public two-way communication of the board members with the larger OSM community on publicly recorded channels has decreased. This can be attributed to several fairly communicative board members having left the board (in particular Frederik, but to some extent also Heather). Additionally it can also be explained by the current board members being more familiar and more comfortable with a sparse and asymmetric top-down style communication in public and perceiving publicity in communication as a burden.
  • The OSMF board also started having closed meetings with external interests, in some cases on a regular basis. This was extremely rare before. We only have limited insight into such meetings in some cases based on minutes published, but they reveal a disturbing level of ruthlessness in lobbying by those external interests that puts into the question the wisdom of having such meetings without public oversight.
  • Most public communication of the board has now the character of after-the-fact presentation of their views on matters where they have already made up their mind rather than a public deliberation in the early phase of a decision making process meant to gather outside perspective before the board members have formed an opinion. In some cases this has gone to extreme forms where the board has released certain highly significant pieces of information just minutes before a public meeting.
  • Public statements of board members also indicate that they view the larger OSM community – or at least critical views from within it – with animosity (example).

Some unexpected positive developments during this year in communication habits in the OSMF we owe to the COVID19 pandemic. This essentially forced changes that were previously suggested already but rejected:

The question is of course how sustainable these positive developments will be or if the powers-that-be will push to restoring previous habits despite their disadvantages of exclusivity and environmental impact.

Social structure of and diversity within the OSMF

One of the first things the current OSMF board did during their term was introducing the so called diversity statement – which i commented on and analyzed in detail and which, as explained, essentially codifies the English language and Anglo-American cultural dominance in the OSMF. Practical effects of this have been relatively subtle so far but there are a few trends becoming visible as to the diversity in the OSMF.

What has happened as a very positive step during the last year is that the board got free membership for active OSM contributors on the way – as mandated by the members in the last general meeting with huge majority. Very late during the year, essentially last minute for applicants to still be able to vote in the upcoming general meeting, but still it did happen and it was apparently quite successful. It is unclear if and how far this changed the membership structure of the OSMF – there have so far only been overall statistics published and no separate numbers on the free membership members. The fact that the application form is available in English language only (though separate translations exist meanwhile) is not a good sign though. In any case, one of the key flaws of the newly established system is that for non-mapping contributions the board decides with practically no external oversight on who may or may not become an OSMF member without paying a membership fee and who, as a result, is among other things able to decide on the composition of the board.

When i look beyond the membership, the OSMF has mostly become less diverse during the past year. That applies to the composition of the board, the working groups and to people who have an influence on what happens in the organization as well as in diversity in work style and work culture. The board is now an all male board with exclusively members from Western Europe and North America – four native English speakers and three Western Europeans (from Germany, Belgium and Luxembourg) all fluent in English. None of them has any cultural background outside Europe or North America.

In the working groups we can observe an increasing difficulty in recruiting qualified hobbyists with diverse backgrounds. As a result of this we saw on the one hand some becoming increasingly dominanted by corporate employees. This is most visible in the Licensing Working Group where the two remaining members who were not corporate employees have left during the past year (Simon and Nuno) and which is now a pure corporate lobbying group (except for Guillaume serving as a board liaison).

On the other hand, we are also seeing a trend of working groups forming increasingly closed and culturally narrow ecosystems within the OSMF, representing relatively narrow and special interests but pursuing them as if they were the interests of the whole OpenStreetMap project. Apart from the LWG, the Local Chapters and Communities Working Group is the main example here. It was established last year and is essentially in pursuit of the mentioned diversity statement but with no commitment to the traditional values and goals of the OpenStreetMap project or with a believable aim to represent the full diversity of local communities in OpenStreetMap (which is obviously impossible in an English language only framework).

A fairly massive trend we have seen during the past year was the establishment of additional ad hoc comittees in the OSMF through the board. The first of those was the diversity committee. The board at first wanted to establish a new working group for that but then realized that it is against established practice for the board to bootstrap working groups. So they created what they called a committee – which is kind of like a working group under direct control of the board. Later added were the microgrants committee (links to details on the members) and the FOSS policy committee. Recruitment of members for these to the outside was based on a public call for volunteers, but actual selection was intransparently done by the board. While it appears like these committees are open to anyone who wants to participate to do so, the board is actually the active gatekeeper here being able to make sure the committees are composed to their liking. Beyond the political preferences of the board (the quote from Guillaume from a different context: People “whose work we know and enjoy” characterizes this quite well) composition of the committees seems roughly based on the diversity statement mentioned above – people have been choosen strictly with a demonstrated good English language communication ability, some effort was put into into female representation and some exotic flavor in form of geographic diversity – but strictly no one with any kind of critical attitude towards the dominating organizational culture in the OSMF. If this was due to selection by the board or due to the lack of applicants i don’t know – but it does not matter. If the board reserves the right to hand pick the composition of the comittees to their liking without transparency or oversight, that obviously makes them unattractive to people with a less adjusted culture and opinion.

So far the committees have not been having a lot of influence, even the microgrants committee has in their selection essentially been overruled by the board, which hand picked several submissions not selected by the committee for separate financial support. But it is clear that the board here is experimenting with ideas how to establish more centrally controlled structures within the OSMF (as opposed to the working groups which traditionally have a high degree of independence). For the upcoming general meeting there is another proposal up to be voted on by the members to establish another kind of committee that is even more tightly controlled by the board and to which the board actually could delegate substantial decision making power.

A significant positive change in this field we saw last year was an increasing number of new local chapters being formally recognized by the OSMF. This is definitely a positive development. The OSMF board has however not taken the logical step to give the local communities a substantial say and formal power in the OSMF. Instead they have engaged in a number of fairly erratic steps: de facto dissolving the Advisory Board, giving local chapters an approximately once per year speaking slot of ten minutes in public board meetings – de facto required to be in English language – and most recently having a fairly ad hoc decision to have an audio meeting with local chapter representatives – which was also in English obviously and therefore visited mostly by native English speakers. It seems essentially the board is stuck here in the commitment to English language and Anglo-American cultural dominance in the OSMF, combined with the de facto impossibility to engage in a meaningful discourse with the local OSM communities world wide without opening up to other cultures and languages.

Checks and balances within the OSMF, applicability of rules

We had prominent failures of the OSMF complying with their own decisions and policy in the past – most notably the Crimea decision in which the board ignored standing OSMF policy to make a politically opportune decision. This year we had an – in my memory – unprecedented number of cases where the board simply ignored their own policy in their decisions and actions. I mentioned some of the more prominent cases in my previous OSMF related blog post already.

What we also saw during the last year was the complete and ostentatious failure of the OSMF in implementing meaningful corruption prevention measures. I wrote about the problem of corruption in and around the OSMF separately. As discussed there, the board has during the last year codified the grossly insufficient status quo in their dealing with conflicts of interests although, both before and afterwards, there have been cases where they failed to meet even these very weak requirements. When they discussed a particularly obvious case of failing to properly address conflicts of interest more recently they not only failed to draw any meaningful consequences from the demonstrated fact that they are not able to reliably recognize even their most obvious CoIs, they also made jokes about this during the meeting, explicitly demonstrating they are not taking the risk of corruption in the OSMF board seriously.

All of these things seem to be symptoms of a larger problem, namely the lack of any meanigful oversight or checks and balances in the OSMF. While the OSMF membership is in principle able to control the board and even dismiss individual board members with simple majority, practically this right is never exercized. Neither is the possibility of the membership to initiate and decide on resolutions in the general meetings to force the board to do certain things – like to follow their own policy. During the past years i have – often to the (at times) strongly articulated displeasure of the board – been by far the strongest critical voice within the OSMF and i have been fairly alone in that – though in individual conversations with others i have had much more differentiated and supportive feedback both from people who agreed with and those who disagreed with my critique. The board essentially has almost zero short term practical incentive to follow their own rules and policy. But, in the long term, the overall credibility of the organization and its ability to have a positive influence on the OpenStreetMap project suffers enormously from that.

Economic changes

The likely most impactful change the OSMF board has made during the past year is the massive increase in planned expenses of the OSMF. Traditionally, the largest part of the expenses of the OSMF is in computer infrastructure. Smaller amounts of money went into trademarks, face-to-face meetings of board members and other things. The SotM conference was also a larger cost factor but also usually brought in at least as much money as it costed. Almost all work in the OSMF, except for part time administrative assistance and specialized work like accounting, was done by volunteers. The costs of this were covered largely by routine donations, earnings through SotM, individual membership fees and targeted donation drives as needed.

Due to the tiered corporate membership system introduced some years ago and several large bicoin donations, the OSMF had towards the end of last year become fairly wealthy relative to the typical yearly expenses and with that background the board has now decided to start spending fairly significant amounts of money on paid work of various kinds. This decision came from within the board – potentially involving external advise but if that was the case it has not been disclosed to the membership. Public consultation on this matter took place – in line with the general pattern i described above – only on parts of the matter (specifically not on the project spendings) and only after the board members had already made up their mind on the topic.

I already commented on the likely social impacts of this move to more paid work in the OSMF. The other important impact of this development is that it massively increases the economic dependency of the OSMF on corporate money. Previously the OSMF was in a financially sustainable sitation, independent of regular corporate contributions, because the necessary spendings could be financed through donation drives and regular donations. This is no more sustainably the case right now and this will worsen in the future because the massive spendings on personell will likely incur additional administrative costs not yet budgeted for.

The effects of this are already visible to the keen observer – large corporate funding contributors of the OSMF, in particular Facebook, have during the past 1-2 years been increasingly bold in ignoring the attribution requirements of the ODbL and the OSMF has during the past year carefully avoided to take any actions of substance in that field. Of course the OSMF board will strongly reject the idea of there being any relationship between their non-action on the attribution matter and the significance of corporate contributions to the OSMF’s financial health in the long term. But the huge economic importance that the good will of the corporations has for the OSMF now is hard to deny. The massive lobbying of corporate representatives in the LWG and in non-public meetings and communications of corporate representatives with the OSMF board speaks for itself.

The important thing to recognize about the major strategic shift in the OSMF of moving from pure volunteer work to at least partially paid work is that it was not argued for through the benefits of it outweighing the disadvantages and risks – a real risk analysis was apparently never made. The main argument was essentially that it is a change without alternatives. Given the likely impacts on volunteer motivation the necessity of this change is of course a self fulfilling prophecy.

Policy changes

A field where relatively little of impact happened last year is in the domain of policy. The board decided on and published quite a large number of internal policy documents but given the slack practice of the board following their own rules these are not much more than momentary expressions of intention and not something that can be relied on in the long term.

The only major development in terms of policy during the last year was the continued drafting of an official guideline on attribution of OSM data use. The LWG (which is as i have pointed out above dominated by employees of corporate data users) has drafted essentially a wishlist of how they would like to see the OSMF to interpret the license in the most profitable way for them in the short term. The board has made some very minor changes to that – which immediately were strongly opposed by the corporate people on the LWG.

The hobby mapper community has rejected both the LWG and the board draft as grossly insufficient, in particular the fact that they – without any basis in the ODbL – intend to allow data users to not display attribution for OSM data use in cases where it is inconvenient for them. I have tried to summarize the views of the craft mapping community in an independent draft for guidance on attribution – which, just like the critical remarks of others in the community, seems to have been largely ignored by the LWG and the board.

At the moment it is not clear what will happen on that front – if the board will take a stand against the hands that feed the OSMF so to speak or if they will accomodate the interests of corporate data users and accept the possibility of an open break with the craft mapping community.

What has also not happened on the policy front is that the board has – despite the need for decisions and actions being fairly clear for more than a year and having been called for by the LWG for quite some time – not taken any steps w.r.t. the upcoming Brexit and the resulting legal and practical operational risks for the OSMF (as a formal British organization but with its main function closely tied to EU database protection legislation). The board investing a lot of time and money into other fields while not doing anything of substance on the Brexit front (at least nothing visible to the membership or the public, specifically they apparently so far have neither decided to prepare for a move nor decided to definitely stay in the UK) is worrying.


Overall, this look back at the past year probably seems like a fairly massive list of grievances regarding the work of the OSMF during that period with only a few bright lights in between. I therefore like to emphasize that for the most part these developments are not the result of malicious intent. As far as i can see the OSMF board is well meaning with many of the changes implemented during the past year i listed above, trying to address issues they perceive to be urgent based on their own perspective and based on the voices within the OSM community they listen to. But, unfortunately, they seem to be unaware of the problematic consequences and risks many of these changes are going to have – due to their own inability to recognize them and the unwillingness to listen to and engage with the arguments and reasoning of those with a more critical view. As i have expressed before, the current board seems to view their public communication mostly as a negotiation of different interests, so critical arguments regarding changes that the board plans are predominantly perceived to be conservative interests that want to keep things as they are for conservatism only and as such are dismissed. It does not seem to be on any of the board members’ radar that the possibility that the full diversity of perspectives in the project (which many of us view both as a challenge and a highly rewarding aspect of OpenStreetMap), is not only valuable but also essential to fill the gaps in the understanding of the social mechanisms within the project each of us (including the board members) has. As a group of seven people often intensely engaged in internal communication among themselves and often dealing with the same matters for weeks, the board has a much increased risk of getting the wrong impression that their collective perception of the project is sufficient to make good decisions without engaging in an open discussion with other views.

In the second part i am going to continue with a lookout on what seem to be likely developments for the coming years in the OSMF, based on what we have seen this year. This is meant to provide more concrete illustration why i see many of the developments of the past year critically. But also it is meant to provide a public test case for my reasoning to some extent – if i am wrong in my predictions i will be happy to admit that i have misjudged the situation. If however i turn out to be right no one can say afterwards that they were surprised by the developments.

October 12, 2020
by chris

Mapping images update

After more than a year without changes i added a number of new images for mapping in OpenStreetMap to my collection. The reason for the gap was that i needed to make some changes to allow the setup to scale better after the tile database had grown to more than 10GB.

What i added so far is:

A new Sentinel-2 image of Ushakov Island – the last larger island discovered on Earth in 1935 – and which due to changing ice margins could use an update in OSM.

Two new Landsat images of Northern Greenland from this year – the first from early July and the second from late July. Both have quite a bit of cloud cover so you need to look which is best in the particular area you want to map. You can also still find the older images i previously added for the area.

An autumn colors Sentinel-2 image from 2019 of the eastern Alps supplementing the one from the western Alps i added before. This is useful for leaf_cycle mapping of forests and in some areas also for updating glacier extents. But beware that there is some fresh snow in October already of course.

An updated Sentinel-2 image of the Vostochny Cosmodrome in eastern Russia from September this year featuring autumn colors in addition to the current status of construction work.

All new images are going to be available in the editors soon probably – if you can’t wait you can also add them manually based on the links provided on my preview map.

October 2, 2020
by chris

Autumn colors 2020

Some impressions from this year’s autumn from a satellite perspective.

Eastern Franz Josef Land in late Autumn 2020

The first one is a glimpse of eastern Franz Josef Land (most prominently featuring Graham Bell Island and Wilczek Land) with a mixture of the last traces of summer thaw in the form of bluish glacier ice and a first hint of snow and frost on the not ice covered areas.

The second is a view from south of the Suntar-Khayata Range in the Russian Far East.

South of the Suntar-Khayata Range in Autumn 2020

In this area autumn colors prominently highlight the two most common tree types in much of northeastern Siberia – Larix gmelinii (Dahurian larch) and Pinus pumila (Siberian dwarf pine). The larch features a yellow-orange color in autumn while the pine stays green.

Larch often dominates in the lower parts of the valleys around the rivers while the pines are more common on the upper slopes near the tree line.

What can also be nicely seen in this view is the naled or Aufeis at higher altitudes in the valleys where river water flooding land areas and freezing at the surface during winter has led to the formation of ice cover that – as can be seen – in some cases does not fully thaw over the summer which is visible in a highly contrasting white on images.

Towards the south tree types become more diverse and the interpretation of the colors less clear. Also the change in colors has not yet progressed that far.

The third and last image is from the Kamchatka Peninula showing the bright white of the freshly snow covered cone of the Kronotsky volcano between the darker autumn colors of the area.

Kronotsky, Kamchatka in Autumn 2020

All images based on Copernicus Sentinel-2 data. For Orientation: Locations of all the views shown can be found in the following map: