Imagico.de

blog

October 31, 2021
by chris
0 comments

The OSMF – last year’s predictions and a look into the future

Last year i predicted a number of trends i identified in the OSMF that seemed to be likely to further develop during this year. I will here have a look at how this turned out so far. Note many of these predictions were a bit more long term than just a single year so it is a bit early for a final assessment of them. If you have not already done so you might also want to look at the previous post where i analyzed the developments in the OSMF during the past year in general.

Autumn impressions near Freiburg

Centralization of the OSMF

This year has shown mixed developments in that regard. With the adoption of Attribution Guidelines designed by the board essentially rejecting significant parts of the LWG proposal the board kind of continued the centralization trend. The creation of new committees where the selection of members was done by the board goes in a similar direction. On the other hand the board has – as mentioned in the first part – so far not really made use of the new instrument of board committees with members that are not part of the board. And as i also mentioned in the first part in my analysis already with the behavior regulation plans the board showed a remarkable reluctance to take any influence or even make suggestions on the work so far. But that does not necessarily disprove my prediction of a tendency for more centralization. Obviously if the work of working groups and committees goes to the board’s liking there is no need to impose top-down influence. An interesting test during the coming year will probably be how independent the newly created EWG will be in their work.

Autumn impressions near Freiburg

Decreasing diversity and brain drain

Last year i predicted that the OSMF would become increasingly a domain of people with economic interests around OSM – either people who volunteer as part of their career or people who get paid for OSMF work. Developments in composition of the working groups seem to confirm this. The newly created EWG seems to consist predominantly of people with an OSM related job and a clear connection between their career/job and the EWG work. The CWG seems now dominated by people with HOT affiliations. And on the newly created Software dispute resolution panel four of five members have OSM related jobs.

Regarding the related trend that the OSMF seems to become culturally more narrow as a result of that – We had in particular a remarkable case in the LWG when the request of Séverin Ménard to join the working group was rejected for fairly questionable reasons which was clearly an act of discrimination because of differences in cultural background and values and personal style and character.

Also matching this trend is the list of candidates for this year’s board election where five out of six candidates have an OSM related job and the sixth (Guillaume) has – through the payment for board work i discussed in the first part – also qualified himself for that category in a way.

This assessment, based on OSM related jobs and OSM related economic interests, is of course only a rough indicator – someone with an OSM related job can in principle decide to volunteer for OSM without that being guided by economic interests. And someone volunteering without an OSM related job might none the less do that to foster their career or to get an OSM related job. But it is still a reasonable rough indicator. And the overall trend that the OSMF is becoming increasingly a domain of people with economic interests around OSM seems to be confirmed quite strongly by it. This is not really that astonishing, given the increased amount of money the OSMF now has and wants to spend. The important thing is that these personal economic interests of people active in the OSMF of course also affect the interests pursued by the organization – more on that below.

Autumn impressions near Freiburg

Seeking influence on OpenStreetMap

This is the trend where i made the most concrete predictions last year so lets see how these turned out:

Community communication channels:

On the behavior regulation front my prediction was spot on i think. The behavior regulation regime is not in place yet but the intention to roll it out soon is clearly there. My guess is what stalls things at the moment is that it is difficult to find people for the moderation team that (a) look somewhat diverse at least on paper (b) are not too obviously unqualified because they are no active users of the mailing lists in question and (c) are willing to take on such a thankless task.

The other front on which the OSMF worked last year towards more top down control and more centralization of OSM community communication channels was the work towards a new centralized communication platform (using the web based forum software discourse) on which according to the OSMFs vision ultimately all community communication the OSMF facilitates is meant to be unified.

Early comments, warning (a) that all the different communication platforms and channels we have and use in the OSM community have very different communication styles and cultures and serve very different purposes and (b) that it would be counterproductive to attempt unifying those into one platform, were quite clearly ignored in the OSMF. As i mentioned last year there are strong voices in the OSMF calling for culturally homogenizing communication in OSM and more top-down management of community interaction. That this ultimately is very likely to backfire – possibly leading to an emphasis of the OSM community separating into distinct filter bubbles with very little substantial communication between them – as i also already indicated last year.

We will very likely see the rollout of both the new unified communication platform and the new behavior control regime next year. It will be very interesting to observe what happens when the community management and cultural homogenization ideas practically collide with the OSM community in all its diversity.

Autumn impressions near Freiburg

The OpenStreetMap website:

So far nothing of substance has happened on that front but the matter has recently been put on the agenda of the board again so it is possible that my prediction was just off by a few months. On the other hand the board would still need to find someone qualified and willing to implement the OSMFs wishes here – which could be quite difficult.

From the board minutes:

the www.osm.org website works well, but there are opportunities for better experience and to grow the community.

That is quite clearly code for we want a more corporate style and less map centered website – hence confirming my prediction from last year. As said there: If that is going to be successful, considering the OpenStreetMap website is traditionally out of scope of the OSMF, is not sure.

Mapping and tagging:

In this point my prediction was clearly wrong or at least much too early (though i mentioned this is a point i was not very sure about). That might be partly because iD development was essentially stalled for the past year so there was no opportunity even for the software dispute resolution panel to make any decisions.

Interesting point on that – the strategic plan outline contains an obscure point called curated tags proposal that might indicate further plans within the OSMF to take influence on mapping and tagging. Can’t be sure about that though.

Because it is unclear where iD development is going in the future i would not want to make a new prediction on this point right now. We will probably see this more clearly in a year or so.

Autumn impressions near Freiburg

In my conclusions last year i also indicated two possibilities for the OSMF to develop into a more positive direction.

The first one (reorganizing the OSMF into a more federated organizational structure with checks and balances and meaningful subsidiarity rules as part of a move outside the UK) is still an option. The board seems to be looking into moving the OSMF. Changing the organizational structure as part of such a move however does not seem to be on the agenda and my reminder that this would be highly desirable did not produce any resonance.

The second one (turning the OSMF into a neutral infrastructure provider for the OSM community under the premise of neutrality and non-discrimination) – given the developments during the past year this seems to have become increasingly less likely – in particular because the OSMF has meanwhile attracted quite significant interests that build on the non-neutral directed money spending under the people whose work we know and enjoy paradigm that is pursued right now. So in short: Sadly that ship has probably sailed.

Autumn impressions near Freiburg

Corporate takeover

There has been some discussion about my blog post from last year and the claim that the corporate takeover has already happened. The problem with my claim of that was that i defined it as the inability of the OSMF to make decisions that are against the interests of its corporate financiers. And proving that inability would amount to proving a negative – which is hard. But, as i discussed in the first part of this post, the strategic plan outline the board has approved this year essentially codifies the alignment to business interests in substance – or in other words: The intention not to make decisions that are against the interests of the OSMFs corporate financiers. And i think a clearly articulated intention makes proving the inability superfluous.

The strategic plan outline has now documented clearly that the OSMF has shifted in their direction from supporting OpenStreetMap as the social endeavor that it is to re-shaping it to the ideal that business interests around OSM in large parts would like it to be – a project with the primary goal being a collection of useful geodata and all social aspects of the project being subordinate to that material goal. It can of course be questioned if this can be classified as a corporate takeover. But the result is similar, the OSMF has over the course of several years mostly adjusted to and adopted the goals and interests of corporations as their own.

And to be clear – this shift in focus is not exclusively for the benefit of large corporations or against the articulated interests of most people around OpenStreetMap. Many people view this shift to be an attractive opportunity for their career or business interests or a chance for a job or some paid work related to OpenStreetMap. And they are not wrong about this. But make no mistake – such small scale interests are just along for the ride. The key influence comes from where the money to satisfy all these smaller interests ultimately comes from.

The long term developments resulting from this shift in direction of the OSMF are hard to predict because they depend mostly on how resilient the OSM community will be against an OSMF increasingly pursuing external economic interests. In the past – essentially for the last ten years from the time when OSM started to become an international cross-cultural social project until today – the OSM community has shown a fairly solid ability to cope with a changing and in parts hostile environment. But that does not necessarily mean much for the future.

The spectrum of possible futures ranges from a quick collapse into what i described elsewhere as a new edition of the International World Map in the digital age using crowdsourcing to scenarios where the core ideas and social mechanisms of OSM essentially continue to work and thrive unaffected by an OSMF in concert with business interests trying to re-invent OSM in their filter bubble. That scenario is essentially based on what HOT has been doing rather successfully for many years. They have made a business out of selling OSM as a platform to perform centrally organized crowd sourced mapping projects and with free maintenance of that data afterwards to aid organizations of various types. The OSMF could try to do a similar thing for the data user market – trying to sell their ideal of OpenStreetMap as a mere collection of useful geodata while the OSM community underneath continues to work under a very different paradigm. It is somewhat doubtful that such a constellation would be stable in the long term but i would not rule it out.

As i have written more than two years ago – no matter how this will turn out, the idea to collect and share local geographic knowledge through egalitarian self-determined cooperation of individuals will survive in the long term because it is something that resonates very strongly with people of very different cultural backgrounds all over the world. The question is only if it will survive within the project that is called OpenStreetMap or outside of it. And maybe a more materialist, more centrally organized and managed OpenStreetMap is what is needed as a counterpoint for this idea to elsewhere continue to thrive and to develop to the next level. In a similar way as the self centered monopolist mapping agencies of the UK and Europe, serving specialized interests but not those of the people, were the necessary trigger for the original idea of OpenStreetMap to form and gain support.

I have during the past year strongly reduced my commentary of the OSMFs day-to-day activities. Main reason is that with an OSMF increasingly acting like a business and serving economic interests providing such commentary increasingly feels like providing free consulting services – which is not what i like doing (i commented more on this mechanism in the past). I will probably continue to observe and analyze the publicly visible activities of the OSMF (for their influence on OpenStreetMap and out of intellectual curiosity regarding the social dynamics within the organization) but i will likely write less spontaneous comments and limit myself mostly to long term afterwards analysis like this.

What i really would like to see in the future is more people from the OSM community coming together and working on projects decidedly outside the sphere of influence of the OSMF. JOSM some time ago made the remarkable decision to reject the offer from the OSMF for financial support and this way making a clear statement that they want to stay independent of such influence. It would be nice to see that kind of becoming a role model for other projects.

As for the OSMFs future – in principle the organization still has the potential to change quite dynamically and to take a more positive direction in the future. With that i mean in particular more enlightened decisions, more based on arguments and reasoning and with a tradition of regular public discourse discussing those, and less based on negotiations of interests as it is done right now. That is most certainly not going to happen next year, given the candidate portfolio in this year’s board election. And with every year of course the influence of external interests in the OSMF gets more strongly solidified, making such a change less likely overall.

Autumn impressions near Freiburg

October 30, 2021
by chris
0 comments

The OSMF – looking back at the past year

About a year ago was the last time i wrote on this blog about politics of the OpenStreetMap Foundation (OSMF) and with the annual general meeting and board elections coming up i think it is time to look back again at the past year and review the predictions i made last year. In this first part i will provide a summary of the most significant developments of the past year from my perspective.

Autumn impressions near Freiburg

Overall developments

Before i get to specific things that have happened during the past year and their analysis i want to describe my impression of the overall trends. Concerning the board the bottom line is that relatively little has changed compared to the year before (where i saw the changes made quite critically in many aspects as you can read up). That means the three board members elected in the end of 2020 (one re-elected who was on the board before and two new members) did not leave much of a distinct signature so far and seem to be standing mostly in the shadows of the more extrovert and more voiceful members whose seats are up for election this year.

That means overall trends that started last year which i described in my analysis back then mostly continued during this year. The plans of the board to expand their own work capacity by creating board committees with additional committee members not from the board so far did not yield any practical results. The board formally created three board committees but only one of them is documented to some extent beyond that (the personnel committee) and there is no documented activity of that so far either.

That also means public activity of the OSMF in general shifted a bit from the board (which massively dominated the previous year) to working groups and special committees this year. This is paired with a further decline in transparency in some parts of OSMF work also outside the board. Several working groups and committees ceased any form of regular reporting to the larger community (LCCWG, Microgrants committee already in 2020, Diversity and Inclusion Special Committee in 2020 and the SotM working group already in 2019) Of many of their activities we only know through reports to the board made during the board meetings (which fortunately are still minuted). Of two newly created committees (Software dispute resolution panel, Special Committee on Takeover Protection) we don’t know anything about their status. Overall the lack of publicly available information meanwhile means we do not only lack knowledge of what the OSMF is actually doing in many cases, we also don’t know if they are doing something or not in large parts. On the positive side i like to mention the newly formed Engineering Working Group which seems to try to be transparent in their work from the start with meetings open to the OSMF members and minutes published in a timely fashion. It will remain to be seen if that level of transparency will be maintained once it comes to making substantial decisions but the will to start with a culture of openness is quite clearly there.

So let me get to a list of key topics that we know the OSMF dealt with or worked on during the past year. A more complete list can be found on the OSMF wiki. I will try here to discuss the most important points only and their political significance and the potential impact on the future of the OSMF and OpenStreetMap.

Autumn impressions near Freiburg

Banking issues

After the OSMF has had already serious trouble in 2020 because their long term bank threw them out there were apparently a multitude of new issues this year. It seems the OSMF had moved to a bank with fairly high fees and to mitigate that used non-bank financial service providers for most of their transactions. And since there was no redundancy in that system of banking and financial services the OSMF was in a pretty bad situation for some time after two service providers blocked their accounts.

What i am not sure about and what worries me is if these two big issues in two consecutive years have lead to introducing serious redundancy into the financial management of the OSMF – in other words: If the OSMF could now survive a similar case (any of their banks or financial service providers freezing their accounts or terminating their contract ad hoc) without running into serious operational troubles as a result. There is nothing in public communication of the board indicating that there have been precautions and redundancies introduced for this kind of situation. I don’t want to assume this not being the case but the question arises of course.

Autumn impressions near Freiburg

The dangerous precedent of paying a board member

In the course of these banking issues the OSMF board made the decision to pay one of them (the treasurer) a rate of EUR 100 per hour for working on the problem. This decision did not gather much attention back then but i consider it highly problematic for a number of reasons:

  • Evidently all board members had a conflict of interest with a decision that sets a precedent for paying a board member for board work. Even if the other board members have no interests in getting paid for OSMF work themselves and will never have such interest in the future (which is something only very few of them can convincingly claim) that does not change anything about this. A conflict of interest is the possibility that there is a secondary interest (the personal financial interest of the board member) colliding with the primary interest the board member is to pursue (the interests of the company). That is evidently the case here (all board members could profit from a precedent in paying a board member for board work). Therefore i think anyone would assume that the decision to start paying board members for board work – no matter if in general or only in specific cases – would be one that can only be made by the OSMF members during a general meeting. And even if that is considered not to be an option the obvious alternative (to hire an outside professional for that work) does not seem to have been considered at all.
  • The financial impact of this decision on the OSMF will in the future probably far exceed the amount paid (and the board as far as i can see still has not informed the membership how much that is by the way). Because this sets a reference point for any future contractor the OSMF might want to hire. If for work to resolve the current blockage of PayPal and TransferWise accounts for which the treasurer has neither a specific formal training qualifying him nor substantial work experience (beyond the 1.5 years as treasurer) the board considers EUR 100/hour an appropriate fee that will in the future likely be a bottom limit for anyone working for the OSMF as a contractor with any substantial formal qualification or work experience in the domain in question. Just for comparison: The people hired for software development work by the OSMF during the past two years (see here, here, here and here) received EUR 50/EUR 62/USD 66 per hour and all had many years of work experience in the specific fields they were working on (and with software development experience being a qualification in very high demand on the market in general). It is unlikely that the OSMF will in the future be able to contract anyone that cheap if the board itself has essentially set the lower bar for work without substantial qualification at EUR 100/hour.
  • On a meta-level (and i realize that many readers probably consider this a too abstract possibility) this decision could in principle incentivize board members in the future to work towards leading the company into precarious situations similar to this one by not creating redundancies and safeguards where that would be prudent and then – if the problem becomes acute – offer to solve it when getting paid. To be clear – i am not saying this specific problem could or should have been foreseen and prevented. But the decision to pay a board member in exceptional circumstances creates an incentive for board members to work towards similarly exceptional circumstances in the future.

This was not the only case by the way of the board giving money to a board member last year. The other case was when the board granted a request by Amanda to finance a Mastodon server she runs. Now you could argue of course this is nothing more than reimbursement for expenses. But it seems quite clear that this request was handled differently than it would have been if Amanda had not been a board member. If not in any other way then at least because someone else asking the board for money for something like that could not have created a circular like Amanda did.

Autumn impressions near Freiburg

Attribution Guidelines

The most visible policy decision during the past year and in particular the one most widely recognized in the OSM community was the adoption of new Attribution Guidelines. I wrote last year that this could be a significant test case for the amount of influence corporate interests have on the OSMF. The board here managed to adopt guidelines that clearly go against the wishes of large corporate data users. I have discussed this in more detail and as written there the board in return gave the corporations a pacifying gift by essentially declaring an exception of the ODbL for use of OSM data in data driven algorithms (a.k.a. training of machine learning models). Independent of that so far the Attribution Guidelines are just a piece of paper. If the OSMF does not act on data users, in particular the large financiers of the OSMF, not complying with those guidelines the decision to adopt them in this form will not be that meaningful. So the board has done a step towards pushing back against the hand that feeds them so to speak. But without more concrete actions this is not that substantial. You could also say the board has essentially delayed the decision if to initiate an open conflict with either the corporate financiers or the craft mapper community.

Autumn impressions near Freiburg

Change in strategic direction of the OSMF

Something that went more below the radar during the past year was the adoption of a so called Strategic Plan Outline. This at the first glance looks like a fairly vague, harmless and non-controversial collection of matters the OSMF has and should have on their radar. But at a closer look it is not, it is essentially a political program outlining a fundamental change in strategic orientation and core values of the OpenStreetMap project and how the OSMF can try to pursue such re-orientation while maintaining the lip service to support but not control OpenStreetMap.

The key to understanding this text lies in the Preamble – which essentially outlines a vision of OpenStreetMap the board has during the past two years increasingly communicated in their statements and policy documents (first noted this in the discussion of the so called diversity statement and further discussed here and here for example) – that OpenStreetMap is viewed primarily as a project to collect useful geodata and no more as a social endeavor and that all social aspects and all other things traditionally considered core principles of OpenStreetMap are subordinate to that goal.

The rest of the document is then essentially just a sketch of a strategy how the OSMF can support OSM turning into a project that follows this vision. That hollows out the support but not steer paradigm at its core because the vision of OpenStreetMap as a materialist project aiming primarily to collect useful geodata through crowdsourced labor goes completely against the core of what OpenStreetMap traditionally stands for and what made the project successful. And the Preamble goes even further – it puts the volunteer mappers contributing highly specific local knowledge on the same level as large corporations contributing satellite imagery and artificial intelligence tools.

So far this text has received no substantial critical reception and discussion in the OSM community.

Autumn impressions near Freiburg

Conflicts of Interest – unwillingness to address obvious problems and impose more substantial regulation

After last year’s events around the Microgrants program have demonstrated clearly that existing Conflict of Interest rules in the OSMF (which rely on people universally being able and willing to recognize their own Conflicts of Interest) are grossly insufficient, an attempt to make the rules more strict at least in some minor ways from within the board has been non-successful. The comments on the circular indicate a continued fundamental lack of awareness of there being a problem at all. This continued also otherwise during this year – the decision discussed above about paying a board member for board work being just one example. The board is currently discussing a second round of the Microgrants program and despite even the committee from the first round (who did not see the need for public oversight on their work in the first round and who decided to ignore the requirement to do all work openly) suggesting in their final report that clearer rules on that are needed, the board in their discussion so far has shown no indication that they see the need for changes from their side or that they even see there is a significant problem at all.

I have written about the whole topic of corruption risks and Conflicts of Interest in the past.

Autumn impressions near Freiburg

Behavior regulation on OSMF channels

One other topic that was fairly prominent during the past year and that i predicted to be a key trend this year was the attempt to impose behavior regulation on communication channels of the OSM community operated by the OSMF. I commented on that quite a bit already.

The interesting thing about this project is that in contrast to other OSMF topics (like the Attribution Guidelines) where the board saw no problem with taking the initiative and requesting changes to document drafts and ultimately writing their own document or the Microgrants program where the board afterwards hand picked several projects not selected by the committee for separate funding this way overruling the committee, the board here seems to very clearly shy away from taking any influence and for the most part from even commenting in substance on the plans and drafts of the self appointed group that developed these.

This is remarkable in my eyes because as i indicated in the above linked text already what i find noteworthy about this project is how uncritically people in the OSMF seem to embrace the totalitarian and culture imperialist tendencies that can be observed around this. My impression is that different people project very different hopes and visions into this endeavor – some see it as a beachhead to ultimately turn all of OpenStreetMap into a managed community where everyone adjusts to certain behavior standards in everything they do and adopts certain cultural values while others view this as a fig-leaf to pacify certain loud voices without changing anything of substance. The bottom line is: No matter how this will further develop quite a lot of people (possibly all) will be miserable.

Autumn impressions near Freiburg

Further Transparency rollbacks

I have already mentioned several more cases during the past year of decline of transparency in the work of the OSMF above (adding to the general trend in pointed out last year already). But there were further examples of transparency rollbacks. As i linked to above it has been established tradition for the OSMF to publish contracts for paid work. This is very important both for oversight over decisions with potentially high impact on the OSMF’s finances but also for fairness between different contractors (equal terms and pay for equal work) and for making sure the OSMF lives up to their social responsibilities towards people working for them. Not to mention that the latent conflicts between volunteer work and paid work are significantly reduced if the contractual terms and payment for paid work are known to everyone.

Now with the hiring of a new iD developer the board seems to intend to break with that tradition and keep the contract secret. Not only that – they seem to have signed the contract without a formal board decision (which likewise would be a first in OSMF history).

Autumn impressions near Freiburg

What is not on the list

In addition to the list of matters the OSMF worked on what is also important to look at are the matters that are missing on that list. During the last general meeting there were three resolutions approved by the members that have not been implemented by the board yet. One was the investigation of paid votes in elections – which the board delegated to a committee in July but where there are no results yet. The other two were related to the membership system – one meaning to allow active contributor members (who don’t have to pay the membership fee because they have contributed to OSM significantly during the past year) to become regular members rather than just associate members. The other to work on membership prerequisites. None of these has been addressed so far. Given the board members are volunteers and there has been significant unforeseen work during the past year as described above it is not advisable to assign too much blame for that to the board. One could of course question the board’s priorities here, especially since they clearly have put items on their own agenda above the tasks they have been given by the membership.

Anyway – the board seems clearly a bit embarrassed by having forgotten about these points and intends to schedule an additional general meeting within the next half year to then decide about a proposal from the board (to be developed until then) on the last point.

While i think more frequent active involvement of the membership in OSMF politics is a good idea, planning to ask the members for approving a resolution on a membership related matter while the previous resolution the members have approved in this regard has not been implemented yet is kind of weird and does not inspire a lot of confidence.

Autumn impressions near Freiburg

Very successful active contributor membership program

To end with a clearly positive point – the active contributor membership program that has been started last year has turned out to be very successful with more than one third of the members now having made use of it. In what ways exactly this changes the membership structure still remains to be seen – the geographic distribution in terms of country of residence does only provide limited information on that. But it has the potential to lead to significant positive effects in the long term.

Autumn impressions near Freiburg

Conclusions

That was my – certainly rather subjective – summary of what happened in the OSMF during the past year.

While last year i had the impression of the board’s actions and decisions being rather erratic, this year – with there not being that many new projects of the board and the OSMF more consolidating its direction – i also have a much clearer view now of the direction in which the OSMF seems to be aiming.

In a nutshell – what the current boards seems to try to do is to re-invent OpenStreetMap in a way that avoids the OSMF having to decide between its financiers, the hands that feed them (the large corporate OSM data users) and the interests of the project. And from a perspective from within the OSMF this makes total sense. Making OpenStreetMap more business compatible by giving it a business like material goal (collection of useful geodata) avoids a conflict that in the long term would have the potential to destroy the OSMF.

I have explained in the past on multiple occasions that i consider the key for OpenStreetMap’s success in connecting people from all over the world across language and culture barriers to be the egalitarian cooperation between individuals sharing their local knowledge. And i think that cannot be substituted with the goal of producing a collection of useful geodata without loosing the social cohesion of the project and the basis of its success. But of course i cannot prove that, i can only describe how my understanding of OpenStreetMap has led me to that conclusion and how – over the years – many further observations of the social mechanisms in OpenStreetMap (in particular regarding how the OSM community tends to deal with conflicts and cultural differences) have confirmed this hypothesis repeatedly so i offer this as an explanation to understand how OpenStreetMap works and why it is successful as a social endeavor.

When the board however believes the opposite to be true, that the core principles of OpenStreetMap can be substituted with or subordinated under the common goal of collecting useful geodata without adverse effects on the social cohesion across language and culture barriers and they act upon this belief to re-invent OpenStreetMap along these lines they essentially put the whole project on the line based merely on their belief regarding the social mechanisms of how OpenStreetMap works – a belief for which they have no corroborating evidence.

But enough for today. In a follow-up post i plan to look back at my predictions from last year and to what extent they came to pass and see if i can make some predictions for the future again.

Autumn impressions near Freiburg

October 10, 2021
by chris
0 comments

Autumn and Spring in October 2021

Together with the two Landsat image visualizations i showed for the Landsat 9 launch featuring northern hemisphere autumn impressions i have added two more Sentinel-2 images to the catalog on services.imagico.de as well.

The first is a fairly clear early spring image of South Georgia showing the beginning of the thawing season.

South Georgia in early October 2021

South Georgia in early October 2021

The second is a nice autumn view of Mount Aniakchak on the Alaska Peninsula in late September with some fresh snow around the rim of the caldera contrasting with the darker surrounding.

Aniakchak in late September 2021

Aniakchak in late September 2021

September 27, 2021
by chris
0 comments

Landsat 9 launch

Today Landsat 9 was launched. As i mentioned before i typically do not discuss Earth observation satellite launches here because the more interesting and practically meaningful date is when the data starts becoming available. But i make an exception for Landsat because of its historic and present day significance for Open Data satellite imagery.

Landsat 8 image recordings for 2020 – see here for more details

Since images from Sentinel-2 started to become routinely available Landsat has kind of lost significantly in attraction to a lot of data users primarily because of the higher spatial resolution and higher revisit frequency of Sentinel-2. However as i have pointed out on various occasions of the two systems Landsat is the only one with a more or less unbiased coverage of all land surfaces of Earth while Sentinel-2 recordings are scheduled based on short term political and economic interests. And with a second satellite providing high quality data and thereby reducing the uniform quality revisit interval from 16 to eight days Landsat certainly is going to get more attractive.

Still of course with Landsat 9 being more or less a copy of Landsat 8 – which is 8 years old already, probably does not seem like the most exciting news to many. What comes after Landsat 9 is still unclear at the moment – there seem to be some studies on possible concepts running at the moment and some not very specific considerations have been published. For quite some time there were indications that Landsat 9 might be the last Landsat producing completely open data and that the US government would look into privatizing the program. With the current US administration giving larger budgets to publicly financed earth observation that seems less likely now but what direction this will go in is unsure as ever. In light of that the boring nature of Landsat 9 offering merely continuity in available data is not necessarily bad news.

Here – to give an idea what to expect from Landsat 9 – two examples from the past few days from Landsat 8.

Mount Katmai and Naknek Lake, Alaska – September 26, 2021

Landsat 8 - Mount Katmai and Naknek Lake, Alaska

Landsat 8 detail

Landsat 8 detail

Northern Greenland – September 13, 2021

Landsat 8 - Northern Greenland

Landsat 8 detail

September 25, 2021
by chris
0 comments

About the lack of progress in quality of satellite image visualizations

Satellite images and their use have become much more popular in recent years. There was already an earlier wave of broader popularization of satellite imagery with image layers being widely introduced in popular map services on the internet between about 2005 and 2010. This for the first time exposed a large number of people to satellite imagery as a source of information and spatial orientation and not just as mere anecdotal photographs.

What i am talking about here however is the more recent trend of the past roughly five years – so about ten years after the first wave. In contrast to the first wave which was primarily about casual use of imagery this second wave of popularization of satellite imagery involves in particular also more serious or semi-serious active use of satellite data by both professionals and hobbyists with diverse (and often not image processing and not earth observation related) backgrounds. Driving this recent trend is in particular the advertisement and PR which satellite operations – both private businesses and public institutions – invest to create interest in the use of their data and services. This combines with many providers of cloud based services trying to attract paying customers for use cases involving satellite data.

Some might disagree with me identifying these two distinct waves of popularization and instead would see a continuous trend. But i see a distinct period of stagnation in popularization between these two phases and i have been actively observing the field over the whole period so i think i have a pretty good read of it.

Anyway – what i want to write about here is not so much these waves of popularization of satellite imagery but how quality of satellite image based visualizations has developed during this. The technical quality of satellites’ image sensors has improved massively over the years – i have written about many examples showing this. Interestingly this trend in satellites is paralleled by a similar (and technologically related) trend in ground based digital photographic technology as well as its use. To illustrate the point i want to make about satellite image visualization i will therefore make a quick excursion into the development of digital photography over about the same time period.

Since the beginning of mainstream digital photography in the late 1990s sensor and camera technology have seen a similar quality development as satellite image sensors. And the use of the image data produced by these cameras has seen a similar development. With that i am not talking about exotic methods employed by a small elite of experts, i am talking about mainstream image processing methods available to and used by many photographers. Based on this development both in sensor and camera technology as well as in processing methodology even a cheap mobile phone camera will produce images out of the box that would have made a professional photographer of the early 2000s using the most state-of-the-art equipment envious. And with some rudimentary learning and training in use of broadly available tools and techniques (either in-camera or in post processing) you can easily improve in the quality of ground level photographic data visualization if you want so far beyond that.

example of the technical level of the quality of ~2005 consumer digital camera technology with low dynamic range, high noise levels and poor color depiction

Example of the technical level of the quality of ~2005 consumer digital camera technology with low dynamic range, high noise levels and poor color depiction

out-of-the-box results from a present day (2020 model) mobile phone builtin camera showing excellent dynamic range and low noise, very decent color rendition (which can be much improved on with larger image sensors available these days) and an equally decent automated visualization for an sRGB color screen

Out-of-the-box results from a present day (2020 model) mobile phone builtin camera showing excellent dynamic range and low noise, very decent color rendition (which can be much improved on with larger image sensors available these days) and an equally decent automated visualization for an sRGB color screen

Getting back to satellite images now – as said the technical development in satellites’ camera technology pretty much reflects the sensor and camera technology development in ground level cameras. But in data processing and visualization technology employed in broad popular use it does not. Since at least the start of Landsat 8 in 2013 – which marked a significant step up in quality of open data imagery available in larger volume – i have been amazed by the stagnation and sometimes even decline of technology and sophistication in processing methods in the visualization of satellite image data. I have written about this previously in a specific case which – through non-sophisticated processing in visualization – excessively undervalued the quality of the image data used. But this is just a single case example of a much broader phenomenon.

Mediocre quality visualization with clipped colors underselling the much better underlying image data

Mediocre quality visualization with clipped colors underselling the much better underlying image data

Better visualization of the same data

Better visualization of the same data

The amazing thing is that this is evidently not a problem of missing innovation because the innovations not used that would be of value to produce much better visualizations have already been developed and are broadly used – in the domain of ground level photography.

A large fraction of the many people working with satellite image data these days and proudly presenting visualizations of such as testimony to their expertise in the domain work without making use of some of the most basic technological and methodological innovations that have been developed for digital image data visualization over the past decade in photography and that are now available routinely in every phone.

Poor visualization of recent Etna activities from Sentinel-3 OLCI data with distorted colors and unnatural color clipping

Poor visualization of recent Etna activities from Sentinel-3 OLCI data with distorted colors and unnatural color clipping

More consistent visualization of the dame data giving a more balanced and more accurate impression of the setting

More consistent visualization of the dame data giving a more balanced and more accurate impression of the setting

So in a way you could for many of the mediocre visualizations you can see these days both on social media and more serious channels essentially comment: The 1990s called, they want their crappy, distorted colors and overexposed JPEGs back.

As for the reasons for the described phenomenon – a huge part of that is probably because people predominantly do not view satellite images as photos but as a more abstract type of depiction.

The reason why even in the most low end sectors of the digital photography market we have over the past two decades essentially had a continuous competitive pressure for improvements in technical quality is because users of these devices and related software assess and judge quality and are able to do so through comparison with their direct personal visual experience. That is not the case with satellite images. To put it very simply: People tend to expect upfront that satellite image visualizations in many ways do not match or are consistent with their personal viewing experience of the areas shown on the images. Therefore they are more inclined to accept badly made visualizations of satellite images.

That does not mean people are unable to appreciate quality in satellite image data visualizations if they are confronted with the difference like in the examples above. So in a way by aiming for excellence in visualization of satellite imagery you show respect for the needs of your recipients without requiring them to articulate those needs as demands.

September 13, 2021
by chris
0 comments

Satellite image news

The title of this section should be interpreted loosely this time – i have not discussed new developments in the domain in general for some time now so some of the news are more than a year old already.

There were a number of of recent and are a number of upcoming satellite launches both in the Open Data domain (in particular Landsat 9 of course) and in commercial Earth observation. In the latter we have had in particular the first two of four Pleiades Neo satellites launched (May 20 and August 16) which are going to offer increased resolution and higher recording capacity compared to the existing two Pleiades satellites. Maxar has been showing their plans for a new constellation of more lightweight high resolution satellites making use of economics of scale in building them and offering more flexibility in recording than single and more heavy system like they are predominantly using right now. But there seems no definitive launch date yet. Still the Pleiades Neo launches evidently will put pressure on them. Planet Labs has also started launching an updated version of their micro-satellites offering additional spectral bands. As usual they have been very tight-lipped in public about this though.

In terms of newly available open data satellite imagery i want to mention in particular the Brazilian CBERS-4A (launched December 20, 2019) and Amazonia-1 (launched February 28, 2021) satellites. It seems the data released by the National Institute for Space Research (INPE) is all available under CC-BY-SA which is more restrictive than the licenses of other open data image providers but still the most notable entry in the open data imagery domain from outside Europe, North America and Japan so far. What is not clear is if the data published by the INPE is all the data recorded by the satellites or if they release only data selected to be published as open data and there is also unreleased data that is not available freely. Considering the data published is mostly limited to South America that is not an unlikely scenario.

Amazonia-1 image example over southern Peru/Bolivia - whole scene

Amazonia-1 image example over southern Peru/Bolivia – whole scene

The Amazonia-1 satellite with its 850km swath and 60m resolution fills a noticeable gap in the European, North American and Japanese systems between the high resolution systems (resolution better than about 50m but swath width less than about 300km) and the low/moderate resolution systems with swath width of more than 1000km but resolutions of 250m or worse.

Amazonia-1 image sample - magnified crop

Amazonia-1 image sample – magnified crop

The actual Amazonia-1 image sensor is not particularly advanced by today’s standards – dynamic range seems to be comparable to Landsat 7 – with the same tendency to saturate the sensor on bright features. It seems to be using 10-bit AD conversion.

Amazonia-1 image sample - full resolution crop

Amazonia-1 image sample – full resolution crop

There also seems to no radiometric calibration data available so far – just the raw sensor values. But still this is a fairly impressive source of imagery available there with the combination of a fairly wide field of view with a moderately high resolution and it would be great to have this available also for areas outside the region in South America covered so far.

Targeting the same gap in available imaging system is another experimental system – SeaHawk – that was launched in 2018 already and that became more prominently publicized recently. SeaHawk is a less than 5kg micro-satellite that records a 216km swath at 120m resolution. That is less ambitious than Amazonia-1 and provides less nominal coverage than a single Sentinel-2 satellite but at that form factor this is pretty impressive, especially also in terms of image quality.

SeaHawk image sample from Eastern Greenland

SeaHawk image sample from Eastern Greenland

SeaHawk image sample from New Caledonia

SeaHawk image sample from New Caledonia

If you look at the images available more closely you can see various issues and flaws, in particular:

  • a significant level of banding and striping due to non-uniformity in the sensor. This could probably be partly compensated by better per-pixel calibration but still this is on a level significantly worse than larger current satellite sensors.
  • a significant decline in resolution towards the swath edges which is clearly the result of limitation of the optics and not of the view geometry (larger viewing distance at the edges – well known for wide angle systems like MODIS and VIIRS) alone.
  • a significant mismatch between the spectral bands leading to color rims at sharp contrast edges.
SeaHawk image sample magnified crop

SeaHawk image sample magnified crop

The most significant operational issue however seems to be the poor out-of-the-box geolocation accuracy of the images. The raw Level-1 data you can download comes without any geolocation data and so far they seem to be manually geocoding the data on the coastline. Micro-satellites like this have only very limited ability to determine their own orientation in space and therefore limited ability to know where exactly they are looking. That lack of accurate information severely limits the ability to properly and accurately geocode the image data. A more detailed discussion of the matter is provided by the operators.

In a quite similar direction of miniaturization goes the PROBA-V plus one project – meant to be a follow-up on the ended PROBA-V – which unfortunately despite being paid with tax money has never been fully open data. It will be a good test case to see if the organizational culture at ESA is changing in light of the Copernicus program (where the open data policy has been imposed from the top by the EU Commission) or if the influential people in and around ESA still prefer to keep the bulk of the data exclusive to themselves and those deemed worthy.

Following up on my posts on Sentinel-3 image use and the severe issues with geolocation data in the SLSTR files – ESA has in late 2019 apparently finally noticed and fixed the problem. Newer files produced (produced after early 2020, indicated by a ‘004’ version string in the file name) do not contain this error any more. Older Sentinel-3B data has also been reprocessed but Sentinel-3A data not yet.

I will follow up on this in a bit more detail in a separate post probably.

Of all the optical earth observations systems in the EU Copernicus program i had not yet looked at the Sentinel-5P satellite. It is of relatively little value for the work i am doing but for completeness reasons i had a closer look. Sentinel-5P features a low spatial resolution hyperspectral sensor which records data in particular about the atmosphere and focuses on the UV and blue domain. It would be great for considering and demonstrating the effect of different spectral band designs of multispectral sensors on visualization of the Earth but unfortunately its spectral coverage has a huge gap right in the middle of the visual range where you would typically have the green band.

Sentinel-5P L1B data false color visualization showing data from Sept 10, 2021 at around 305nm, 390nm and 475nm

Sentinel-5P L1B data false color visualization showing data from Sept 10, 2021 at around 305nm, 390nm and 475nm

The illustration above shows the visualization of the data of a single day in the UV-Range and in the blue spectrum in a false color display. The strips along the flight direction of the satellite are caused by the different scattering characteristics depending on illumination and view direction. A large part of the signal in the ultraviolet is caused by scattering in the atmosphere. Different gases and aerosols have different scattering and absorption characteristics depending on the wavelength – visible on that day in particular in the mid west of the USA where the smoke from fires shows up in strong red-brown tones. The Sentinel-5P sensor offers very detailed spectra of every pixel recorded. Overall more than 150GB of L1B data were processed for the production of this illustration (and that is just three of eight spectral bands).

Sentinel-5P by the way is the only satellite from the Copernicus program that covers the whole planet surface during a single day.

Finally a bit of seriously bad news – NASA has last year announced that the Terra satellite is nearing the end of its life and will – due to the lack of fuel (i discussed this in more detail for EO-1 in the past) – start to drift in its orbit. It has meanwhile exceeded nominal specifications for the recordings and will probably reach a 15 minute difference in timing/position of its images by September 2022 and will likely cease producing usable images by 2025 latest.

I will probably write a bit more about the Terra satellite and its significance for the historic development of Earth observation from space as well as its present day significance. In particular i would also like to discuss the MISR instrument. Most people know Terra in particular for the MODIS and ASTER sensors and MISR – being somewhat in the shadow of the more popular MODIS is one of the most underrated and underused Earth observation systems currently available.

I have also updated my diagram of Multispectral Earth Observation satellites with many of the systems mentioned above and a few others as well as some updates and fixes for existing data.

Multispectral Earth observation satellite sensors - September 2021

Multispectral Earth observation satellite sensors – September 2021

September 3, 2021
by chris
1 Comment

Green Marble version 2.1

I am pleased to announce an update to my global satellite image product Green Marble.

The Green Marble is a medium resolution whole earth image offering an unsurpassed uniformity in quality across the whole planet surface on both water and land coloring, based on recent data and a hundred percent cloud free.

The Caspian Sea depicted by the Green Marble 2.1

The Caspian Sea depicted by the Green Marble 2.1

The version 2.1 i introduce here is an update of the data basis used for water rendering, which is generated from Sentinel-3 OLCI data. The data basis for that has been more than doubled compared to version 2, meaning that more than 100TB of Sentinel-3 data were downloaded and processed for the production of this image.

This broader data basis resulted in a significant reduction in artefacts and noise levels in water surface rendering, which demonstrates the excellent convergence characteristics of the processing methods used.

Contrast enhanced visualization of the noise level reduction in depicting open ocean areas

Contrast enhanced visualization of the noise level reduction in depicting open ocean areas

To give some idea of the volume of data processed – here a visualization of the number of ultimately useful data samples (likely water, likely free of cloud, sea ice and sun glint) that went into processing.

Number of usable samples of water color data processed in production of the Green Marble 2.1

Number of usable samples of water color data processed in production of the Green Marble 2.1

Here a few examples of comparison between version 2 and version 2.1 illustrating the improvement in quality.


North Sea example (large: GM 2/GM 2.1)


Indonesia example (large: GM 2/GM 2.1)


Korea example (large: GM 2/GM 2.1)


South Sandwich Islands example (large: GM 2/GM 2.1)

The reduced sea ice coverage in the new image is the result of some adjustments made to processing and the additional water data used and does not reflect a reduction in actual sea ice coverage during the past years.

I updated the preview map on maps.imagico.de where you can look at the image in more detail. The product page on services.imagico.de is updated to the new version as well. If you are interested in using this image in your own project you can contact me through the form on the product page or via email. Here some additional samples featuring the improved water rendering.

Torres Strait

Torres Strait

Great Barrier Reef

Great Barrier Reef

Svalbard

Svalbard

Caribbean Sea

Caribbean Sea

August 31, 2021
by chris
2 Comments

Testing client side map renderers on polygons

I have in the past commented critically on the vector tiles hype that seems to engulf and dominate much of the digital cartography sector these days. What i dread in particular is the really saddening effect this trend seems to have in some cases on cartographic ambition of map producers – the most iconic case example being maybe swisstopo – whose traditional topographic maps are by many considered one of the pinnacles of high quality map design with uncompromising cartographic standards and ambitions – and which now sinks to the level of presenting something as crude as this as a serious contribution to their digital services.

One of the core issues of this trend in my eyes has been that it has been fueled largely by economic factors (in other words: cost cutting of map providers) while practically across the board being executed on the engineering level in an highly superficial and sloppy way. If you look at software development projects in the domain you can almost universally see the short term economic goals in the foreground and a lack of capacity and/or willingness to thoroughly analyze the task from ground up and design and develop a solution with a decent level of genericity and scalability and to invest the time and resources a decent solution calls for.

But while i am confident that this critique is justified i realize that for many people working within the vector tiles hype so to speak it is hard to follow these critical considerations without specific examples. And i also want to better understand myself what my mostly intuitive critical view of the vector tiles trend and the technological developments around it is actually caused by and better identify and where possible quantify the deficits.

Starting at the basics

What i will therefore try to do is exactly what software engineers in the domain seem to do much too rarely: Look at the matter on a very fundamental level. Of course i anticipate that there are those who say: These tools are meant to render real world maps and not abstract tests. Why should we care about such? Well – as the saying goes: You have to crawl before you can learn to walk. To understand how renderers perform in rendering real world maps you need to understand how they work in principle and for that looking at abstract tests is very useful. That you rarely see abstract tests like the ones shown here being used and discussed for map rendering engines in itself is indicative of a lack of rigorous engineering in the field IMO.

And i will start at the end of the data processing and rendering toolchain – the actual rendering – because this is where the rubber meets the road so to speak. We are talking about map rendering here which ultimately always means generating a rasterized visual representation of the map. And despite some people maybe believing that the vector tiles trend removes that part of the equation it of course does not.

In this blog post i will look at the very fundamental basics of what is being rendered by map renderers, that is polygons. The most frequent visual elements in maps that rendering engines have to render are polygons, lines, point symbols and text. And a line of a defined width is nothing different from a long and narrow polygon. Text contains curved shapes but they can with controllable accuracy be converted into polygons. Not all renderers do this in practical implementation, some have distinct low level rendering techniques for non-polygon features and there are of course also desirable features of renderers that do not fit into this scheme. So i am not saying that polygon rendering ability is the only thing that matters for a map renderer but it is definitely the most basic and fundamental feature.

As you can see if you scroll down i only look at simple black polygons on white background examples and ignore the whole domain of color rendering for the moment (color management in map renderers or the lack of it is another sad story to write about another day). Because of the high contrast this kind of test very well brings out the nuances in rendering quality.

The difficulties of actually testing the renderers

I am looking specifically at client side map renderers – that is renderers designed to be used for rendering online maps on the device of the user (the client). This creates several difficulties:

  • They typically run in the web browser which is not exactly a well defined environment. Testing things on both Chromium and Firefox on Linux i observed noticeable differences between the two – which however do not significantly affect my overall conclusions. The samples shown are all from Chromium. I will assume that the variability in results to other browsers and other platforms are similar. Needless to say that if the variability in results due to the client environment is strong that is a problem for practical use of these renderers on its own. After all as a map designer i need predictable results to design for.
  • They are only the final part of an elaborate and specialized chain of data processing designed for cost efficiency which makes it fairly tricky to analyze the renderer performance in isolation without this being distorted by deficits and limitations in the rest of the toolchain.

I managed to address the second point by running the first tests based on GeoJSON files rather than actual vector tiles. All three renderers evaluated support GeoJSON input data and this way you can reliably avoid influences of lossy data compression affecting evaluation of the rendering performance (which is – as i will explain later – a serious problem when practically using these renderers with actual vector tiles).

The test data i created, which you can see in the samples below, is designed to allow evaluating the rendering performance while rendering polygons.

Test patterns used for evaluating polygon rendering

Test patterns used for evaluating polygon rendering

I will quickly explain the different components

  1. Adjacent polygon test with four touching rectangles that are a single multipolygon geometry (which is technically invalid – but all renderers accept this). At two different rotations.
  2. Adjacent polygon test with four touching rectangles that are separate geometries. This is the classic case where AGG produces artefacts (as explained in detail elsewhere).
  3. A Siemens star like test pattern showing resolution, aliasing and color mixing biases.
  4. Dent pattern to show lines of various width both positive (black on white) and negative (white on black) features. Shows any bias between those two cases, artefacts in narrow feature rendering and step response at a strait polygon edge in general.
  5. Inverse Siemens star like pattern with narrow gaps of variable width testing geometric accuracy in rendering and geometry discretization artefacts.
  6. Test for rendering uniformity on circular arcs. Contains both a narrow black line as well as a white gap of same size for checking equal weight of those.
  7. Fine comb gradients testing color mixing and aliasing artefacts.
  8. Fan comb for testing aliasing artefacts.

The test subject

I tested the following client side renderers

  • Maplibre GL – the FOSS fork of Mapbox GL, which was essentially the first browser based vector tiles renderer and can probably be considered the market leader today (though with the fork and development lines diverging and becoming incompatible at some point this could change).
  • Tangram – originally a project of Mapzen this is the other WebGL renderer that has significant use. Having been started by Mapzen its orientation was a bit less narrowly focused on short term commercial needs but it is essentially a ‘me too’ response of Mapzen to Mapbox without much own ambition in innovation (which was a bit of a general problem of Mapzen).
  • Openlayers – the veteran of web map libraries took a very different approach and is now a serious contender in the field of client side map renderers. It uses Canvas rather than WebGL.

For comparison i also show classic Mapnik rendering. This is essentially meant to represent all AGG based renderers (so also Mapserver).

All examples are rendered in a size of 512×512 pixel based on GeoJSON data (in case of the client side renderers) or from a PostGIS database in case of Mapnik.

What serves as reference is – as explained in more detail in my previous discussion of polygon rendering a supersampled rendering – the brute force approach to rendering – performing plain rasterization at a much higher resolution (in this case to represent all the fine details in the test pattern i chose 32767×32768 pixel – that is 64×64 supersampling). When you size that down to 512×512 pixel using plain pixel averaging you get the following.

Supersampled rendering using equal weight pixel averaging (box filter)

Supersampled rendering using equal weight pixel averaging (box filter)

In terms of general precision and edge acuity this is the gold standard. However plain averaging of the pixels is prone to aliasing artefacts as you can observe in the tests for that. Other filter functions can be used to reduce that – though always at some loss in edge acuity and potentially other artefacts. Here the results with the Mitchell filter function – which massively reduces aliasing at a minor cost in edge acuity.


Box filtered reference (left) in comparison to Mitchell filtered rendering

Test results

Here are the results rendering the test geometries in their original form (as GeoJSON data).

Mapnik


reference (left) in comparison to Mapnik rendering

Mapnik is – as you can expect – fairly close to the reference. That includes similar aliasing characteristics. On the fine comb gradient it is even better since it precisely calculates the pixel fractions covered while supersampling does a discrete approximation.

The main flaw is the coincident edge artefacts you can observe on the tests for that on the lower left. Notice however these only occur with coincident edges between different geometries. Within a single geometry (tests on the upper left) this problem does not manifest (though practically using this as a workaround does likely have performance implications). The coincident edge effect also has an influence on the narrow gaps of the inverse Siemens star on the right. In other words: This does not depend on edges to be exactly coincident, it has an influence to some extent anywhere where several polygons partly cover a pixel.

Maplibre GL

Maplibre GL is interesting in the way that it has an antialias option which is off by default and without which the renderer produces results too ridiculous to show here. So what is shown here are the results with antialias: true.


reference (left) in comparison to Maplibre GL rendering

Even with the antialias option the results are bad. Primarily because of a massive bias in rendering enlarging all polygons by a significant fraction of a pixel. This is visible in particular on the fine comb gradients which are turned completely black and the dent pattern where the narrow gaps on the bottom are invisible while the thin dents on top are more or less indistinguishable in width.

In addition to this bias – which turns many of the test patterns non-functional – diagonal edges show significantly stronger aliasing (stair step artefacts), narrow gaps between polygons are non-continuous in some cases and fairly non-uniform on the circle test.

Tangram


reference (left) in comparison to Tangram rendering

Tangram does not have the bias Maplibre GL shows, positive and negative shapes on polygons are rendered in a matching fashion. But it shows similar levels of aliasing which produce a significant amount of high frequency noise even on strait lines and smooth edges like the circle test. Thin lines are rendered proportionally in weight but non-continuously. And the comb gradients produce massive artefacts.

OpenLayers


reference (left) in comparison to Openlayers rendering

OpenLayers is very different in its technical basis of rendering and this shows in the results. In many ways it is the closest to Mapnik and the reference rendering and it shows the least edge aliasing artefacts of all the client side renderers – and accordingly also the clearest representation of the circle test. It however also has its issues, in particular it shows the coincident edge artefacts similar to AGG based renderers (both on the separate and combined geometries), it has issues in proportionally rendering narrow polygon features – including some bias towards the dark (positive) lines compared to the bright (negative) lines.

With actual vector tiles

So far i have shown test results of vector tiles renderers not using actual vector tiles – to look decidedly at the rendering quality and precision and not at anything else. But the question is of course: Do they perform this way, for better or worse, also with actual vector tiles.

The question is a bit tricky to answer because vector tiles are an inherently lossy format of storing geometry data. Coordinates are represented as integers of limited accuracy or in other words: They are rounded to a fixed grid. In addition most vector tile generators seem to inevitably perform a lossy line simplification step before the coordinate discretization and the combination of the two tends to lead to hard to predict results.

The PostGIS ST_AsMVTGeom() function for example (which is meant to be a generic all purpose function to generate geometries for vector tiles from PostGIS data) inevitably performs an ST_Simplify() step without the option to turn that off or adjust its parameters.

So what i did first for testing if there is any difference between how GeoJSON data and vector tile data is rendered is to generate very high resolution vector tiles (extent 65536) as reference. However as it seems only Openlayers seems to actually work with this kind of high resolution vector tiles directly, both Maplibre GL and Tangram seem to reduce the resolution on the client side (the motivation for which eludes me). Hence evaluation of these renderers is limited to normal low resolution tiles.

Client side coordinate discretization in overzoomed display in Maplibre GL (left) and Tangram (middle) in comparison to Openlayers

Client side coordinate discretization in overzoomed display based on the same vector tile data in Maplibre GL (left) and Tangram (middle) in comparison to Openlayers (right)

Here are examples generated with the PostGIS ST_AsMVTGeom() default extent of 4096 – in comparison to the GeoJSON tests shown above.


Maplibre GL rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)


Tangram rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)


Openlayers rendering from geoJSON data (left) in comparison to vector tiles data (extent 4096)

As you can see the differences are small and mostly in those areas where aliasing artefacts are strong already. So the principal characteristics of the renderers are the same when used with vector tiles compared to uncompressed geometry data.

What this however also shows is that vector tiles are an inherently lossy format for geometry representation and that there is no such thing as a safe vector tiles resolution beyond which the results are indistinguishable from the original data or even where the difference does not exceed a certain level. Keep in mind that the test patterns used are primarily rendering tests meant to show rendering performance. They are not specifically intended as test patterns to evaluate lossy vector data compression. Also it seems that the resolution of 4096 steps when generating tiles is pretty much on the high side in practical use of vector tiles and the level of lossy compression using ST_Simplify() performed by ST_AsMVTGeom() is less aggressive than what is typically applied.

I will not go into more detail analyzing vector tiles generation here and how rendering quality is affected by lossy compression of data. Doing so would require having a decent way to separate coordinate discretization from line simplification in the process.

Conclusions

Doing this analysis helped me much better understand some of the quality issues i see in vector tile based maps and what contribution the actual renderer plays in those. With both the lossy data compression in the tiles and the limitations of the rendering playing together it is often hard to practically see where the core issue is when you observe problems in such maps. I hope my selective anaylsis of the actual rendering helps making this clearer also for others – and might create an incentive for you for taking a more in depth look at such subjects yourselves.

In terms of the quality criteria i looked at here, that is primarily precision, resolution and aliasing artefacts in polygon rendering, the tested client side renderers perform somewhere between so-and-so and badly. Openlayers shows the overall best performance. Tangram is on the second place, primarily because of much more noisy results due to aliasing artefacts. Maplibre GL makes the bottom end with a massive bias expanding polygons beyond their actual shape essentially rendering many of the tests useless and making any kind of precision rendering impossible – while being subject to similar levels of aliasing as Tangram.

Do i have a recommendation based on these results? Not really. It would be a bit unrealistic to make an overall recommendation based on a very selective analysis like this. Based on my results i would definitely like to see more examples of practical cartography based on the Openlayers vector tiles rendering engine.

If i should really give a recommendation to people looking to start a map design project and wondering what renderer to choose for that it would be a big warning if you have any kind of ambition regarding cartographic and visual quality of the results to think hard about choosing any of the client side renderers discussed here. Because one thing can be clearly concluded from these evaluations: If we compare these renderers with what is the state of the art in computer graphics these days – both in principle (as illustrated by the reference examples above) and practically (as illustrated by the Mapnik rendering – though it is a stretch to call this state of the art probably) – there is simply no comparison. Is this gap between what is possible technically and what is practically available in client side renderers warranted by engineering constraints (like the resources available on the client)? I doubt it, in particular considering what is behind in the web browser and how resource hungry these renderers are in practical use. But i can’t really tell for sure based on my limited insights into the domain.

July 12, 2021
by chris
4 Comments

State of the Map 2021 – observations

The last weekend the State of the Map conference 2021 took place – the second time as a purely virtual conference, you can find my comments on last year on this blog as well.

This commentary is somewhat preliminary because i did not yet view some of the most interesting talks because of real world obligations and the multiple competing tracks. And since the talks were pre-annouced to be recorded and to be made available afterwards while the panels, workshops and self organized sessions were not i focused on the latter a bit more.

The concept of a virtual conference – second iteration

I presented some general musings on the idea and appeal of a virtual conference in last year’s comments most of which still applies of course. Many of my more radical suggestions to depart from a pure 1:1 virtualization of an in-place conference and embracing the fundamentally new chances of the virtual format have not been followed. I have the impression that many of the influential people in the OSMF and around the conference still view the virtual conference as a temporarily necessary stop-gap replacement of what they actually want – a physical meeting of the international OSM jetset – and therefore don’t want to start things that could in any way be incompatible with moving fully back to what they know and enjoy.

Compared to last year where everything had a fairly improvised feel to it, this year the whole conference framework was much more slick and organized. But to me this is not really a positive thing. It is not that last year things technically did not work on a larger scale and this year i saw plenty of cases of (minor) glitches, in particular when you tried to join a room the browser tab often froze for a considerable amount of time. To me this slickness of this year’s conference was most visible in form of the much tighter constraints, less freedom, more builtin distance between the participants of the conference and less freedom for free form interaction. Specifically:

  • I missed the pads that were last year used for comments, questions and discussions of the talks. The free form and non-linear nature of the pads offered so much more freedom for interaction, working together and expressing yourself than this year’s linear chats and isolated fire-and-forget questions.
  • Linear group chats in general pose a structural disadvantage to slow typers and non-native English speakers and incentivize asocial tactics to dominate a discussion. They promote shallowness in discourse and essentially prevent any more elaborate discussion with multiple participants.
  • Last year’s SotM was essentially an open event where everything (streaming of talks, interactive conference rooms, pads for discussion) was openly accessible to everyone without registration. I saw this as a huge benefit and it contributed a lot to the open get-together flair of that event. This year everything (except for maybe streaming of talks) was only open to registered visitors of the conference (and i am not even sure you could still spontaneously register during the conference).
  • There does not seem to be a permanent record of the chats available to follow up on and continue afterwards.

Don’t get me wrong, this was still overall a well made conference and i saw a lot of effort on side of the organizers to make this suitable and inviting for a large variety of people (including in particular the translation efforts – which i did not check out myself). But from my perspective it was in many ways also a step back compared to last year and i kind of fear that is in a way a (potentially mostly subconscious) step to pave the way back to a primarily non-virtual conference again.

Observing filter bubble maintenance in action

The most memorable impression for me was observing the echo chambers most of the panels at this conference turned out to be and essentially observing filter bubble maintenance in action.

The panels i am talking about here are:

First of all there might be a bit of a cultural difference of course. To me a panel discussion on any serious topic would be an event where people with significant differences in their perspective and opinion on a matter meet, each presenting their views and then discussing them so they come to insights that represent an additional value compared to the sum of its parts. In other words – to me a panel discussion is essentially a dialectic format.

The panels from this conference however were more like the panels of a fan convention where actors and producers of popular performing arts praise themselves and each other for their achievements, tell anecdotes and engage in friendly and shallow banter and the main aim is for everyone to have a good time and enjoy themselves.

Now don’t get me wrong – my issue is not that such events took place at SotM. What i found remarkable however is how the participants of all of these panels clearly thought that they were having a meaningful and insightful conversation while what happened was essentially that they reaffirmed each others’ existing preconceptions. The main result of the events was the reinforcement of the existing filter bubbles of the participants and the parts of the audience on the same wavelength by excluding huge parts of what is already a small and relatively homogeneous subset of the OSM community (the conference visitors) from their perception and celebrating a friendly conversation within that bubble to be a discussion with meaning regarding and representative for the larger reality.

All of these panels had participants that could have provided very valuable contributions to a dialectic or argumentative panel discussion on the subjects discussed (and where i would for example definitely say i could learn a lot from having a conversation with them on the subject in question). But without anyone on these panels questioning the preconceptions and assumptions of the participants and opening their eyes to other perspectives on the matter the intellectual value and the gain in insight of these events for listeners on the subject matter was negligible.

In the Paid-Editing/AI-Mapping Tools panel – which was moderated by a professional scientist who should by profession be used to and experienced in questioning and challenging his views and preconceptions i decided to inquire about this with the question:

Why does your panel consist exclusively of people working for large corporations. Do you think others (small businesses, self employed people, hobbyists) have nothing important to contribute on the matter of paid editing and AI mapping tools or did those you have asked decline to participate?

The answer was sobering: They indicated not to believe that those not represented on the panel in their eyes had nothing important to contribute but they still chose to only invite representatives of large corporations.

Some thoughts on the map reading Q&A

When the call for proposals for this conference was announced i was fairly unsure if to submit something. The most natural thing to submit would have been a talk. But the talks were – like last year – to be pre-recorded to then be streamed to the visitors at the specific times of the schedule only. It seems a bit weird to me – not to say disrespectful to the international audience residing in different time zones – to impose such an artificial constraint (since the talks are prerecorded) and to essentially imitate a live event without it actually being live.

Now having talks pre-recorded and not live makes a lot of sense, in particular it lowers the hurdles to have a talk at such a conference massively. Last year i suggested to make use of the natural possibilities coming with that and to make the talks available to be watched by the visitors whenever is convenient for them in advance and to then asynchronously discuss them and to wrap that up with a live discussion similar to what we had now. But this suggestion was not picked up, probably because it was perceived to be a too radical change and it would be incompatible to moving back to a live non-virtual conference without giving that up again of course.

Anyway – my thought was if the conference organizers want to have the conference as a live event only i would want to do something that actually uses the benefits of live interaction with the audience. At the same time i wanted to try introducing the idea of asynchronous participation (like in my suggestion of making the talks available for viewing in advance) into what i do. That is how i developed the idea of an interactive Q&A workshop.

Here are my spontaneous observations and impression of how it went:

For the pre-event asynchronous involvement of the audience i had created a wiki page and a pad to submit topics to discuss. Speaking live at a virtual event of this form is difficult because there is an almost complete lack of real time feedback from the audience. At the same time it is live so in contrast to a pre-recorded talk you can’t just make cuts and re-record certain parts or the whole thing if necessary. Because i kind of anticipated that my original plan was there to be more audience interaction during the workshop.

But while i had advertised it on the diaries and on the OSM-Carto issue tracker there was only one submission (From Jerry (SK53), many thanks for that) before the start of the conference. And Jerry was not present live in the workshop (which was fine – turning up at the workshop was not a prerequisite for submitting case examples) so i could not start off with an interactive approach to the whole thing. And although during the conference a number of more submissions came in i ended up not having any live dialog in audio/video with the audience. That was a bit unfortunate (and i could probably have tried to encourage that more) – it would likely have been an improvement for both me and the audience to connect a voice and a face to the questions and have more symmetric discussion of the case examples.

In the end most of the interaction took place on the chat and the questions. The splitting into chat and questions in the conference interface was something really confusing and IMO quite pointless. You could only show either the chat or the questions and it is easy to miss additions to one of them while concentrating on the other. I see that there is a potential benefit of having anonymous questions but even that would seem possible while still conflating the two in the same display.

Over the duration of the workshop (which ended up to last for 1:30 instead of just one hour) the interactivity improved (probably mostly because i got more used to keeping an eye on the chat and questions and integrating them into my discussion) but for my taste it was still too asymmetric – as said i would have liked more true dialog.

The other important observation i made: For a live event like that involving a screenshare presentation it would be really good to have a triple-screen-setup. You would essentially need:

  • one screen with the conference interface (video, chat, controls etc.)
  • one screen to share
  • one screen for your notes and staging of materials to share

I only had two screens set up connected (had a third one – but that was on a different computer so of limited use) which made things a bit more complicated.

In terms of the actual content of the workshop – it ended up being a bit less basic map reading and more advanced map design questions – like problems of scale dependent rendering and how to deal with the rural-urban divide. I had prepared some material on the matters of road layering and urban landuse, which was more focused on basic explanation of the map but which i ended up not using because there were more than enough questions from the audience to fill the time. But i am a bit afraid that for some of the less experienced listeners the whole workshop was a bit too advanced towards the end.

What i am kind of wondering is if it would make sense to develop something like a regular recurring event out of this outside SotM. Considering how widely the standard map style is used and referred to in the OpenStreetMap world it is quite definitely severely under-explained and under-discussed on a level above the basic “I want feature x to be rendered” and outside the scope of actual style development. Any input on that idea would be welcome.

Navigating the Maze

May 12, 2021
by chris
10 Comments

Navigating the Maze – part 2

This is the second part of a two part post on roads rendering in OpenStreetMap based digital rule based map styles. In the first part i have explained and analyzed the roads rendering system of OSM-Carto and its issues, here i want to explain and discuss the re-designed road rendering system i drafted for the alternative-colors style and the improvements and new features i implemented in it.

A big warning upfront: This road rendering framework is essentially just a sketch. I will explain in the end what i hope in the long term eventually will come out of it. At the moment this is – as said – a bit sketchy and i don’t want to make any claims that it is overall better by any metric than the existing one in OSM-Carto. Or in other words: It is as much a study on what is possible as much as it is one for what is not. Keep that in mind when you think about actually using this. Also if you work with this technique or in general with the AC-style you should be sure to turn off JIT optimization in the Postgresql configuration. Because that does not deal well with the kind of queries used here apparently.

Rethinking layered rendering

The technical limitations of the OSM-Carto road rendering system i discussed in the first part are to some extent caused by specific design choices in OSM-Carto. But for the most part they are the result of principal limitations of the toolchain used by OSM-Carto. The most fundamental limitation of that is that the order in which you query the data from the database and how this is split into a lot of separate queries is directly tied to the order in which things are drawn on the map. There are some limited options to interfere with that on the raster compositioning level (OSM-Carto is using that a bit already and i use this more in the alternative-colors style – a topic for a different post) but in the end you are pretty fundamentally stuck with this paradigm of query order and drawing order being tied to each other.

Most of the software OSM-Carto is using – specifically Carto and Mapnik – are no more actively developed. That means there is not much of a chance that anything will change about this constraints. This also means that the only major component of the OSM-Carto toolchain that is still actively developed is Postgresql and PostGIS.

This essentially suggests the most promising path for rethinking the roads rendering is to work around the limitations of the layered rendering framework as described by combining all road rendering components into a single layer. If you look at the layer list in the first part with 16 layers you can see that this is a pretty radical step. Fortunately there are already quite a few pre-existing changes in the alternative-colors style that make this a bit simpler. Some layers i have dropped (tourism-boundary), others i had moved outside the road layers range (cliffs), which leaves the following layers to deal with:

  • tunnels
  • landuse-overlay
  • line-barriers
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing
  • highway-area-fill
  • roads-fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom
  • waterway-bridges
  • bridges

In addition i moved the aerialways layer above the bridges (because practically most aerialways will probably be physically above bridges they cross) – although this could in the future be changed and integrated back with taking the layering into account of course. That is still 13 layers of course.

So why would i want to combine all of this in a single layer? This has a number of advantages. Since one layer means one SQL query only you don’t have to query stuff several times from the database and you can avoid duplication of SQL code much better. At the same time addressing some of the fundamental limitations of the OSM-Carto road rendering, in particular the lack of proper layering for highway polygons, becomes much easier and does not require a significant volume of additional code.

And most importantly you don’t have to deal with the limitations of the various methods to define the drawing order i discussed in detail in the first part and the complex and difficult to understand interaction between them because you can simply put the features in the order you want to render them in using ORDER BY in SQL. That means you cannot use group-by and attachments any more of course and for every use of attachments you have to essentially duplicate the features to be drawn in SQL (like using a UNION ALL query). But practically that is relatively simple and adding an ORDER BY statement to the wrapping query of the single roads layer is a much more intuitive and better maintainable way to do this in my opinion.

Layering in SQL

So how does the query combining all these 13 layers and the improvements i implemented look like? Before you go to looking through the more than 1700 lines of SQL code you can have a look at the basic structure where i stripped most of the actual queries, in particular the lengthy column lists which make up most of the query. A more elaborate sketch of the structure is available as well. The query makes fairly extensive use of common table expressions to query different sets of features and then use them in different contexts. On the base level of the query (roads_sublayers) we have the CTEs

  • roads_all – which contains all the linear road features from tunnels/bridges/roads-casing/roads-fill as well as things derived from them (for new features – i will discuss those later),
  • road_areas_all – which contains all the road polygon features from highway-area-casing/highway-area-fill
  • tc_all – which contains the stuff from turning-circle-casing and turning-circle-fill

These CTEs currently are independent of each other. roads_all basically contains the UNION ALL from the existing road layers combining highway lines and railway lines – plus the alternative-colors specific features, which use a number of additional CTEs that reference each other.

Based on the three main CTEs there is a stack of 16 sublayer queries combined with UNION ALL. You can find the non-road layers from the above list in there while the road layers do not match the legacy layers directly. Ignoring for the moment those sublayers connected to new alternative-colors specific features we have:

casing
background
fill
area_casing
area_fill
tc_casing
tc_fill
waterway_bridges_casing
waterway_bridges_fill
line_barriers
ferry_routes
landuse_overlay

We don’t need separate sublayers for tunnels and bridges because the sublayers are not what primarily defines the drawing order. The drawing order is defined by the ORDER BY statement that follows the sublayer stack.

That means across all sublayers with a non-null int_bridge attribute (that is everything except the non-roads layers) the features with int_bridge=yes are rendered on top in the order of their layer tag. It does not matter what these are, whether road line bridges, highway polygons, turning circles or waterways. Same for tunnels at the bottom. Apart from that the ordering is similar to the one in OSM-Carto – although i made some adjustments, the effects of which are however mostly very subtle.

Map design in SQL

The most important result of this change is that it allows consistently layering all bridge and tunnel features – no matter if linear roads, highway polygons or waterway bridges – according to the layer attribute as explained in the first part. Here the layering in comparison between OSM-Carto and the alternative-colors style:

Linear roads and waterway bridges layering in OSM-Carto (left) and AC-Style (right)

Highway polygons layering in OSM-Carto (left) and AC-Style (right)

The other fundamental thing i changed and that was also an important prerequisite for the new features i added is to define the line widths for drawing in SQL. This has a two-fold purpose: First it is to make these settings available in SQL for geometry processing and second: To avoid the error prone repetitive way of defining the drawing widths for the different zoom levels as shown in part 1. Parts of this change were already implemented in previous changes where i introduced implicit embankment rendering. This is currently based on a YAML file where you define the different line widths. This file is processed by a python script into alternatively MSS code or an SQL function containing nested CASE statements for selecting the line width based on zoom level and road type. This method is not yet set in stone – an alternative would be to have a separate table in the database containing the line width values. I have now extended approach to also include the casing width values in addition to the overall line width.

Ground unit line width rendering

One thing this change facilitates is allowing to more easily implement ground unit line width rendering at the high zoom levels. The motivation for that is the following: At the low zoom levels roads are universally rendered with a line width larger than their real physical width on the ground. That is universally the case in maps because otherwise the roads would not be visible at smaller scales and even when they would be that would not make for a well readable map. There is however a transit at some point where the drawing width or the roads becomes less than the physical width of the roads. That transit typically happens – depending on latitude, road class and the specific road – at zoom levels between 16 and 18. Drawing the roads more narrow than they are physically is however typically not of benefit for map readability and it also conflicts with the general aim you have when designing large scale maps to move to less geometric abstraction and a depiction closer to the geometric reality on the ground.

Unfortunately you cannot address this simply by increasing the nominal drawing width of the roads in general because not all individual roads of a certain class have the same width of course and what you definitely don’t want at the high zoom levels is to draw the roads wider than their actual width and then obscure other content of the map next to the roads. And fortunately quite a few of the roads in OpenStreetMap have a physical width specified – two million (or about 1 percent) and even more (12 million or 6 percent) have a lanes tag that allows estimating the likely physical width of the road with some accuracy.

To be able to draw lines in ground units on the map you have to be able to convert between map units (also called Mercator meters), ground meters and pixels on the map. The conversion factor between ground units (as tagged with the width tag for example or as estimated from the number of lanes) and map units is called the scale factor of the projection and unfortunately there is no support for directly getting that in PostGIS. I used a generic function that allows approximating that:

ST_DistanceSphere(
  ST_Transform(ST_Translate(geom, 0, 1), 4326),
  ST_Transform(geom, 4326)
)::numeric from (select ST_Centroid(!bbox!) as geom) as p

I do this based on the bounding box of the metatile – not so much for performance reasons but because that ensures geometry processing within a metatile is using the same scale factor – which is essential in some cases. At the zoom levels in question the scale factor will never vary significantly across a metatile. One thing to consider of course in the future is to – for Mercator maps – replace this generic function with the simpler formula specifically for the Mercator projection (because that function is used quite a lot).

The conversion between map units and pixels on the map by the way is the factor !scale_denominator!*0.001*0.28 – so in total you have

pixels = ground_meters/(scale_factor(!bbox!)*!scale_denominator!*0.001*0.28)

This formula is used in carto_highway_line_width_mapped() – which is implemented in roads.sql. Overall the rule for the line width of roads is as follows:

  • If a width is tagged use that as the ground width.
  • If no width is tagged but the number of lanes is tagged then estimate the overall ground width based on the road class and (conservatively estimated) the typical lane width for that type of road.
  • If the ground width determined that way is larger than the nominal line width set by the style than draw it in that width, otherwise in the nominal width.

Currently the style still contains double drawing of the roads in both nominal and ground width (essentially determining the maximum graphically) because i initially had them rendered in different styling. But that should be removed in a future version.

Here is how this looks like at z17 with various different tagged width values – in comparison for different latitudes to show the effect of the variable scale.

Ground unit rendering of roads at z17 at 40 degrees (left) and at 60 degrees latitude (right)

This applies for the roads drawn with the standard casing + fill style. For airport runways the formula is varied to estimate the likely width based on the runway length (using a formula initially suggested for OSM-Carto). For road features with a simple solid/dashed line signature (paths/tracks) the ground width is only visualized at z18 and above – using the styling of footway and track polygons (with some adjustment for tracks to avoid the appearance being too heavy) in addition to the centerline signature. And this ground unit visualization is only used when the drawing width in ground units is large enough to be meaningfully depicted in addition to the centerline.

The ground unit rendering of roads however is not technically as simple as it might seem because of another implicit assumption made by the roads rendering system in OSM-Carto, namely that the drawing order of road classes with the same layer as defined by z_order is in sync with the drawing width progression. That is of course no more universally the case when the tagged width is depicted which would lead to artefacts like the following.

Road junction with artefacts

Solving this requires modifying the road geometries in those cases which is kind of a hassle to do.

Road junction without artefacts

Lanes visualization

Since i already made use of the lanes tagging for estimating the ground unit width of roads i decided to visualize the number of lanes at the high zoom levels as well. As you can see this follows the implicit rules documented which say that trunk roads and motorways are assumed by default to be two lanes. lane_markings=no is indicated with dashed lines instead (click on the image to see).

Lanes visualization at z18+

Lanes visualization at z18+ (click to see also the unmarked version)

In practical use cases ground unit line width rendering and lanes visualization looks like this:

Roads in Strasbourg at z18

Roads in Heidelberg at z18

Roads in Freiburg at z19

This one demonstrating the unfortunate and semantically nonsensical practice of German mappers to tag every road under a bridge as a tunnel.

Of course this style of visualization for both the lanes and the tagged width of the road does not work too well when the number of lanes and the road width changes. This requires mapping and interpretation of additional data, in particular what is mapped with placement=*, which is currently not taken into account.

Turning circles & co.

While we are on the matter of drawing width or road lines – i also extended the existing visualization of turning circles and mini-roundabouts from OSM-Carto with distinct symbolization of both of these as well as turning loops and passing places.

turning circles, turning loops, mini-roundabouts and passing places at z17

turning circles, turning loops, mini-roundabouts and passing places at z17

All of course working in combination with the bridge layering and variable width depiction. The diameter tag for turning circles is also taken into account.

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads ...

turning circles etc. in combination with specified ground width or roads and tagged diameters, bridges etc. for residential roads …

... as well as tracks

… as well as tracks

Sidewalks

The next new feature i looked into visualizing is implicitly mapped sidewalks and cycle tracks. Like in case of embankments sidewalks can be mapped both implicitly as an additional tag on the road (sidewalk=both/left/right) as well as explicitly separately as a way with highway=footway and footway=sidewalk. Both are widely used and there is no clear preference among mappers and it is therefore a bit problematic that OSM-Carto only renders the explicit sidewalks. Like in case of embankments the explicit version is easier to render in a primitive fashion while good quality visualization is much easier with the implicit mapping because you can more easily take the relationship between the road and its sidewalk into account.

What i implemented is a demonstration how visualization of implicit sidewalk mapping as well as that of implicit cycleway tracks with cycleway:right=track/cycleway:left=track can work. This is shown at z18 and above

Rendering of sidewalks and cycleway tracks

Rendering of sidewalks and cycleway tracks

and works also in combination with bridges, fords and implicit embankments. They are not rendered on tunnels to avoid obscuring ground level features.

Rendering of sidewalks in combination with various road variants

Rendering of sidewalks in combination with various road variants

And here is how this looks like in real world situations.

Sidewalks in Ladenburg at z18

Sidewalks in Ladenburg at z18

Sidewalks in Portland at z18

Sidewalks in Portland at z18

Steps details

Rendering of roads in their ground unit width also provides more space to visualize additional details on steps. The normal drawing width of steps does not really provide enough room for showing more information without affecting the recognizably of the steps signature. Here is what is shown with nominal drawing width and when the drawing width is large enough at z18+ for wider steps tagged with their width.

Steps details at z18+

Steps details at z18+

Unpaved roads

The distinct depiction of paved and unpaved roads is one of the oldest open issues in OSM-Carto. The difficulty is that it has to work in combination with a lot of other road properties already visualized. Various forms of dashing have been tried and have been found either to be very noisy in appearance or too similar to existing depictions of tunnels (which use a dashed casing), construction roads (which use a dashed fill) or access restrictions (which use a dashed centerline on the fill) to be intuitively understandable without affecting map readability in general. The most promising idea so far has been the use of a fine grained structure pattern for the road fill. That is highly intuitive in its meaning and can be tuned to be fairly subtle. Lukas Sommer has developed this idea some time ago to be ready for use but unfortunately Mapnik did not support 2d fill patterns for line features and while this was added as a feature some time ago this is not yet available in many production environment and especially not in release versions of Kosmtik. And workarounds for this are a bit ugly to implement in the OSM-Carto road rendering system. I have now added this to the AC-style by generating polygon geometries for the unpaved roads using ST_Buffer() and rendering those instead of the normal line features. That is easily possible with the new setup since the line widths are available on SQL level anyway.

The basic styling of the unpaved road rendering used

The basic styling of the unpaved road rendering used – click to see at z18 with and without tagged width visualization

It is important to consider how this looks like in combination with various other road properties – which you can see in the following illustration. There are some issues with the readability of the access restrictions – which is mostly due to their depiction offering very different levels of contrast on the different fill colors. That needs a closer look at and either tuning the access restriction color or revisiting the principal design of this feature in general.

Unpaved road styling in combination with other road attributes at z17

Unpaved road styling in combination with other road attributes at z17 – click to see more

Also here a few examples of practical situations with this styling.

Unpaved roads

Unpaved roads

Unpaved pedestrian area

Unpaved pedestrian area

Open ends

That’s it for the main new road features and improvements i introduced within the new road rendering framework. There a quire a few other tweaks improving consistency in rendering – like getting tunnels and bridges rendered for raceways. And a subtle centerline on runways and taxiways. And as already hinted at there are quite a few open ends in the whole thing design wise that could use further work. But this blog post is already longer than it should be so i will close for now.

Performance of this new roads rendering approach in terms of computing resources required is not bad by the way. While the roads query is often the slowest query of the style – which is not that astonishing because it contains the data previously processed by a lot of different queries – it is not exceedingly slower than other queries. And it is often faster than the water barriers query, which has been the slowest so far and which is why i added an option to disable that for faster rendering.

Technically one important thing left unfinished is that there is currently a lot of duplication of rendering rules between SQL and MSS code. Like for example i have introduced several zoom level selectors in SQL to limit certain queries to specific zoom level ranges (of the form WHERE z(!scale_denominator!) >= 18) which are unnecessarily mirrored by a matching selector in MSS. This is redundant and prone to errors so it is important to come up with some clear principles which parts of the style logic should be in SQL and which in MSS code.

I updated the sample rendering gallery to the style changes by the way and also added quite a few new samples there.

Conclusions on part 2

The main outcome of the whole endeavor of re-designing the road rendering system this way from my perspective is that

  1. yes, there are much better ways to structure rendering of the roads in an OpenStreetMap based map style without taking shortcuts and as a result failing to consistently representing the road topology than the classic OSM-Carto road rendering setup.
  2. The ad hoc implementation of this in the way presented here based on the Carto+Mapnik toolchain is anything but ideal. In particular the difficulty of debugging the SQL code is a major concern.
  3. The map rendering framework that provides a decent platform for implementing the kind of rendering of roads presented here quite clearly still needs to be developed.

My main hope is that what i show here provides valuable additional insights into what exactly the key factors are to be able to develop map design in an efficient, flexible and scalable way (and scalable here is meant from the perspective of style development, not from the side of rendering efficiency – which is an important matter too of course – though pretty much a separate issue). At least to me it seems after working on this project i now have a much better idea of what the core requirements would be for a future map rendering and design system suitable for sophisticated and high quality rule based cartography with zoom level/map scale and styling specific geometry processing.

In terms of the new features implemented it is too early to draw any definitive conclusions i think. Most of them, in particular the variable width ground unit rendering and the lanes depiction i consider to be in a highly experimental state with a lot of open issues – both technically and design wise. The question of what constitutes the core of meaningful OpenStreetMap based cartography for the very high zoom levels (z19+) that goes beyond simple extrapolation of lower zoom level designs plus functioning as a POI symbol graveyard and that does not depend on mappers essentially directly hand drawing the map is still largely undiscussed.

Navigating the Maze

May 11, 2021
by chris
1 Comment

Navigating the Maze – part 1

This post is part of a longer series on map design questions. I have not written much more recently on my work on the alternative-colors style and there is a bit of a backlog of topics that deserve some discussion. I will make a start here with the most heavy of all – the roads. In this first part i am going to explain the OSM-Carto road rendering system and analyze its issues and limitations.

The challenge of depicting roads

The without doubt most complicated part and for any OSM-Carto map style developer the most serious challange when working on OSM-Carto or one of its derivatives is the roads layers. As Paul some time ago compactly put it the roads rendering in OSM-Carto alone is more complex than many other map styles are in total. Roads (and in context of OSM-Carto roads by the way generally refers to all lines and other features along which land navigation of humans takes place – including roads for automobile navigation, paths for navigation by foot, animal or other vehicle as well as railways and traditionally also runways and taxiways on airports, which however have more recently been moved outside of the roads layer system in OSM-Carto) can be complex in their semantics, connectivity and above all in their vertical layering. Representing that in a well readable and accurate way in a map is a challenge to put it mildly.

Most simpler map styles take shortcuts in this field – which tends to show up in map in non-trivial situations. See for example the following case example:

Road example in Dallas, US in OSM-Carto

Road example in Dallas, US in OSM-Carto

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

Same example in other maps: left side from top: Bing, Esri, TomTom, right side from top: Maptiler, Mapbox, Thunderforest

All maps have their flaws in roads rendering – OSM-Carto in particular with the labeling – which however is a different matter that i am not going to cover here. But beyond that it offers by far the most consistent display of the connectivity and layering of the roads in this intersection.

The way this is done in OSM-Carto i will describe in a moment – but before i want to emphasize that OSM-Carto producing a consistent rendering of how the roads are mapped in OpenStreetMap is of high importance for the quality of data in OSM. It helps ensuring that the information that is necessary for an accurate interpretation of the data is mapped and that the reality is not recorded in a distorting fashion to achieve a certain rendering result. More than in any other field of mapping in OpenStreetMap probably the rules of mapping roads have been developed in connection with how the data is used in rendering while still maintaining the principle that what is mapped and tagged represents verifiable elements of the geography and not just prescriptions how things are supposed to show up in the map. This is an accomplishment that cannot be overstated and that is important to maintain and extend in the future.

The OSM-Carto road layers

To understand the way OSM-Carto renders roads it is important to first understand how the drawing order is defined in the toolchain used by OSM-Carto (Carto and Mapnik). In decreasing priority the drawing order is defined by the following:

  1. The layers of the map style: Data that is rendered in the map is queried from the rendering database in the form of distinct and separate layers and is rendered in exactly the same order. The layer stack is defined in project.mml and starts with the landcover layers (at the very bottom) and ends with labels.
  2. The group-by parameter of the layer in question (in case it is present): The way this works is that the query of the layer is ordered primarily by one column of the results table and by specifying this as group-by parameter you tell carto/mapnik to group the features by that parameter and render them kind of like they were in separate layers.
  3. The attachments in MSS code. That is the ::foo selectors. Those kind of work as a method to accomplish layered drawing within a single layer. But in contrast to the group-by parameter every attachment needs to be coded explicitly, you cannot automatically base attachments creation on some attribute in the data.
  4. The order in which the data is queried in SQL (typically through an ORDER BY statement)
  5. The instances of the drawing rules in MSS code: Those are the prefix with a slash you put in front of the drawing rules. Like a/line-width: 2; b/line-width: 4; Such rules lead to several lines being drawing, one immediately after the other, on a per-feature basis.

OSM-Carto contains exactly two layers that use the group-by feature: The tunnels and the bridges layers for the roads. Overall the roads related layers (and other layers in between) look like this (from bottom to top):

  • tunnels (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill
  • landuse-overlay (military landuse overlay)
  • tourism-boundary (theme park and zoo outlines)
  • barriers (line barriers like fences, hedges etc.)
  • cliffs
  • ferry-routes
  • turning-circle-casing
  • highway-area-casing
  • roads-casing: with attachment ::casing
  • highway-area-fill
  • roads-fill: with attachments ::halo (low zoom only), ::fill
  • turning-circle-fill
  • aerialways
  • roads-low-zoom: with attachment ::halo
  • waterway-bridges
  • bridges (with group-by layernotnull): with attachments ::casing (high zoom only), ::bridges_and_tunnels_background (high zoom only), ::halo (low zoom only), ::fill

To better understand the terminology used – here an illustration for how the different parts of the road signature are called.

Road signature elements at high zoom levels (z18)

Road signature elements at high zoom levels (z18)

Road signature elements at low zoom levels (z11)

Road signature elements at low zoom levels (z11)

All the individual road line layers (tunnels, roads-casing, roads-fill and bridges) are ordered in SQL primarily by layernotnull (which is the layer tag with an implicit default value of zero assumed) and by z_order – which represents the hierarchy of road classes.

If we ignore most of the non-road layers for the moment and focus on the high zoom levels we have essentially the following drawing order:

  1. tunnels – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)
  2. casings of normal non-tunnel, non-bridge roads – ordered by
    • first turning circles, then polygons, then linear ways
    • layer
    • z_order (+ some other criteria in SQL)
  3. fills of normal non-tunnel, non-bridge roads – ordered by
    • first polygons, then linear ways, then turning circles
    • layer
    • z_order (+ some other criteria in SQL)
  4. waterway bridges – ordered by
    • layer
  5. bridges – ordered by
    • layer
    • casing, background, fill
    • z_order (+ some other criteria in SQL)

This is how this looks like visually:

Layering of linear roads in OSM-Carto

Layering of linear roads in OSM-Carto

That ordering alone however – despite its complexity – does not ensure a consistent depiction of the roads network. That also relies on the specific way the road lines are drawn. To understand this have a look at the following illustration.

Bridge drawing principles with different layers connecting

Bridge drawing principles with different layers connecting

It shows self intersecting road loops as either tunnels or bridges. On the left how this is rendered when mapped as a single way. To actually represent a road crossing over above itself in form of a bridge or tunnel rather than intersecting with itself on the same level you have to split the way into two and assign different layer values.

That the two ways still visually connect where they meet at the top with their ends depends on the way they are drawn with flat line caps for the casing and round line caps for the fill. This leads – as you can see in the rightmost version – to some smaller artefacts when the bridge/tunnel way ends meet at an angle. That brings me to technical mapping rule number one that is good to consider in mapping:

Bridge or tunnel ways connecting at their ends should always have collinear segments where they meet.

The other thing that i hope this explanation shows is the meaning of the layer tag in OpenStreetMap. The wiki is fairly detailed on this but also a bit confusing. What the layer tag does is describing the relative vertical position of the physical object the feature represents, in particular a road, relative to other elements in the immediate vicinity that it is not connected with. It does not define the drawing order in the map (which depends on how exactly the map shows different features), it does not document the vertical position of a feature relative to ground level. This leads to the second technical mapping rule you can take away from this

Use the layer tag sparsely (both in the extent of where you apply it and regarding what cases you apply it in) where it is necessary to avoid ambiguities in the description of the 3d topology of what is represented by the data.

In all other cases there are other tags that will most likely better describe what you want to document – like bridge=*, tunnel=* location=underground, level=*. Or you are trying to do something you should not do (like – as a mapper – drawing the map yourself).

Limitations of this approach

Some limitations of the described system of road drawing are already visible in the samples shown. Others are not and i will have to explain those in the following. Before i do that however i want to mention one additional aspect of the OSM-Carto road rendering system that i have not mentioned so far and that frequently is an additional source of difficulty for developers.

The actual drawing rules for the roads in CartoCSS for large parts serve the different layers together. In particular for example the styling of the roads fill for tunnels, ground level roads and bridges is using the same CartoCSS code. That has the advantage that the different styling variants (see image below) are consistently applied to all the layers. And changes made to that do not need to be duplicated at several places. But it also makes analyzing and understanding the code overall more difficult because after the road features are split into the different layers for rendering them in the proper order these layers are tied together into a meta-maze again for styling.

Different styling variants on all the road classes - many of these can be cross-combined

Different styling variants on all the road classes – many of these can be cross-combined

Beyond that the main principal technical disadvantages of the OSM-Carto road rendering system are:

  • It requires quite a lot of SQL code being duplicated into the different road layers, in particular the tagging interpretation that is being done on SQL level. Some of this has been reduced more recently using YAML node anchors and references – specifically the roads-casing and roads-fill layers as well as the turning-circle-casing and turning-circle-fill layers. That only works when the SQL code is exactly identical though.
  • This of course also means that it requires unnecessarily querying the same data several times from the database. Practically this is not a big issue performance wise but still it is not a very elegant way to do this.
  • Any non-trivial improvements and extensions to be made to the road rendering system typically require additional layers or at least adding to the SQL code that then needs to get synchronized across several layers and adding to the already significant complexity of the whole system.
  • If you look at roads.mss you can find a lot of fairly mechanical repetitions of code to implement the variation of styling parameters across zoom levels. Like the following:


    [zoom >= 13] { line-width: @secondary-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19; }

    Often this is even repeated for other layers or variants – like in this case with:

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @secondary-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @secondary-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @secondary-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @secondary-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @secondary-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @secondary-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @secondary-casing-width-z19; }

    [zoom >= 13] { line-width: @secondary-width-z13 - 2 * @major-bridge-casing-width-z13; }
    [zoom >= 14] { line-width: @secondary-width-z14 - 2 * @major-bridge-casing-width-z14; }
    [zoom >= 15] { line-width: @secondary-width-z15 - 2 * @major-bridge-casing-width-z15; }
    [zoom >= 16] { line-width: @secondary-width-z16 - 2 * @major-bridge-casing-width-z16; }
    [zoom >= 17] { line-width: @secondary-width-z17 - 2 * @major-bridge-casing-width-z17; }
    [zoom >= 18] { line-width: @secondary-width-z18 - 2 * @major-bridge-casing-width-z18; }
    [zoom >= 19] { line-width: @secondary-width-z19 - 2 * @major-bridge-casing-width-z19; }

    That adds a lot of code volume and is prone to errors. It also makes styling changes that go beyond simple adjustments of the parameters in the CartoCSS symbols used complicated.

On the level of currently present functionality some of the most significant limitations of the current system are (and note these are only those related to the main road layer stack as described – things like road labeling are a different story):

For the polygon features rendered (that is primarily highway=pedestrian and highway=service polygon features) there is no support for bridges and tunnels. That means the above grid example with highway=pedestrian and highway=service polygon features looks like this:

Layering of highway polygons in OSM-Carto (or the lack of it)

Layering of highway polygons in OSM-Carto

Solving that would be possible but you would have to integrate the polygons into all the linear road layers and that would significantly increase their complexity (and the amount of code that needs to be duplicated between them). Same applies for turning circles obviously – although this is a much less significant limitation practically of course.

Another limitation is that while the waterway bridges are correctly layered among themselves they are not interleaved with the road bridges. So a waterway bridge is always drawn below a road bridge independent of the layer tags.

Road layering with waterway bridges in OSM-Carto

Road layering with waterway bridges in OSM-Carto

Practically this is not a big limitation of course (how many waterway bridges are there on Earth where there is a road bridge below?).

What is more important is that for avoiding complexity OSM-Carto has some features not within the road layers that should be included because they can be above or below road features in reality. Specifically that is bus guideways (which have always been separate creating various issues) and airport runways and taxiways (separated more recently) – which is rarely a problem but there are cases where it is (though in most cases of enclosed pedestrian bridges spanning taxiways the bridge is purely mapped as a building and not as a footway).

What i have not covered are the frequent concerns about overall drawing order of roads, in particular road polygons, above other map features, in particular buildings and landcover features. That is a whole other subject and one connected to much more fundamental question about map design and mapping semantics in OpenStreetMap. And it is not something related to the specific design of road layer rendering used by OSM-Carto.

Conclusions on part 1

I tried above to explain how OSM-Carto renders roads and how this system – while being one of the most sophisticated road rendering systems you can find in OSM data based maps – has its limitations. Anyone who tries to implement improvements to map rendering in the style in any way related to roads struggles with those. In the course of my work on the alternative-colors style i have now decided to try re-designing the roads rendering in a fairly fundamental way. How this looks like and what new features and improvements can be achieved with that will be subject of the second part of this post.