Thomas Skowron's Blog

Load Balancing Without Giving Away the Keys

Mon, 31 Aug 2020 15:49:00 +0000

Distributing the load of a web application or especially an API endpoint using a load balancer (LB) is highly useful: You can get better performance, make software rollouts smoother and withstand node failure.

If an application node becomes unavailable due to failure or a planned maintenance, the LB notices that it does not respond and stops send traffic to it.

The LB itself can go down, too, but since LBs can be made stateless, failover and redundancy can be achieved more easily through e.g. the usage of CARP¹ or anycast. Many cloud providers offer managed load balancers.

Since HTTPS has become the standard even for non-private data transfers, the management of TLS certificates and keys needs to be considered as well. Often software operators opt to decrypt the TLS on load balancer level and send traffic via unencrypted HTTP to the nodes, compromising confidentiality.

TLS can also be handled by the nodes itself, but this bears two challenges: All the nodes need the same key and certificate and also, if you’re acquiring the certificates from Let’s Encrypt or any ACME-compatible CA, they need to serve the appropriate authorization challenge.²

For serving HTTPS directly from a Go application, there is the certmagic library, which also powers the automatic TLS management in Caddy. By default, it manages key generation, cert acquisition and wraps the HTTP server with TLS. The storage for the keys is rather flexible: Normally they’re stored on disk, but you can add a different adapter or write your own. There is a third-party S3 adapter, which allows you to store your certs/keys/challenges on Amazon S3. If you configure your nodes to use the same S3 bucket, it doesn’t matter which node will be asked by the load balancer to serve the challenge, all of them can answer, because they’re pulling the challenge via API.

Personally I found it unacceptable to store all the keys in plain text on some else’s hard drive. Perfect forward secrecy prevents from decoding HTTPS traffic before a breach, but with a third party service it’s impossible to determine whether such an event took place.

This is the reason why I built a custom certificate storage for certmagic, which stores the keys, certs and challenges on any S3-compatible store (e.g. Backblaze, Digital Ocean Spaces or your own Minio instance), but encrypts it using NaCl’s secret box³ before sending it off.

You don’t have to trust any of the provider’s claims about ”at rest“ encryption and you don’t have to implement any highly available storage yourself. You just have to spend a few pennies per month for any S3 storage, without lock-in.

This allows you to do HTTPS without having to break encryption mid-way and you can offload TLS to the nodes, instead of the LB having to do all the work.

FreeBSD Manual on CARP ↩︎
To be pedantic, only if you’re using either TLS-ALPN or HTTP authorization. You can also use DNS validation, if your DNS software or zone provider supports this. ↩︎
More specifically, the secretbox implementation in Go ↩︎

Printing with SwiftUI

Wed, 08 Jul 2020 16:12:00 +0000

This post combines one of the world’s oldest information technologies with one of the newer ones: Printing on paper and SwiftUI. SwiftUI is Apple’s rather new framework to declaratively design user interfaces, which runs on all their platforms. It has a few rough edges, but it is very powerful, because it allows mixing and matching with classic NSViews on macOS, so you can even use them for print (or PDF) output.

macOS has always had rather great printing support and for a first-class Mac app this is a feature one shouldn’t miss.

Integrating the print workflow into a new app is not too hard and starts with connecting the “Print…“ menu item’s action to an outlet. The menu configuration is not part of SwiftUI, but lives by default in a dedicated storyboard Main.storyboard.

With an option click drag (aka right click drag) you can connect the print menu entry to an outlet in your AppDelegate class.

Your delegate will look something like this:

@IBAction func applicationPrint(_ sender: NSMenuItem) {
...
}

Inside this method you’ll need to setup your view that you will want to print, set up the Print layout info and then hand over to the user to make a decision about the output format and device.

First of all, inside your delegate, set up an NSPrintInfo instance and configure it:

let printInfo = NSPrintInfo()
printInfo.scalingFactor = 0.7

I used a smaller scaling factor, because the font sizes suitable for screens were too large for printing in my particular case.

Unless you want to print the whole window with e.g. all input elements and text fields (in which case you can simply pass window!.contentView to the print routine), you need to initialize your printable view and populate it with some data. Let’s assume you have a SwiftUI view called EntryListView and it takes a list of string as data:

let entryListView = EntryListView(data: ["This is a string", "And another one", "One more"])

Since this is a standalone SwiftUI view, it needs to be contained in an NSHostingView:

let printContainerView = NSHostingView(rootView: entryListView)
printContainerView.frame.size = CGSize(width: 800, height: 600)

This view is now wrapped and can be passed to a print operation alongside the print info configuration:

let printOperation = NSPrintOperation(view: printContainerView, printInfo: printInfo)
printOperation.printInfo.isVerticallyCentered = false
printOperation.printInfo.isHorizontallyCentered = false
printOperation.runModal(for: window, delegate: self, didRun: nil, contextInfo: nil)

If the user selects “Print…” from the menu bar or hits “⌘P”, a print dialog will popup with your view ready to be printed:

Migrating Foreign Keys in PostgreSQL

Thu, 21 May 2020 08:30:48 +0000

Applications often need to work with external IDs, e.g. UUIDs of third party services or SKUs. In such cases, an external ID should be stored just once in a mapping table and from there on only referenced by an internal (e.g. serial) foreign key. Without a dedicated mapping table, you will carry such external identifiers through all tables. This may seem reasonable at the beginning, but may soon cause trouble, e.g. if you want to add a second type of external service or when you realize, that those seemingly unique identifiers are not as unique as assumed.

Even though I should know better, this kind of mistake happens to me from time to time anyway. Or it happens to others and I have to save the day.

Because PostgreSQL is not the wild west, if you reference a field in a different table, this will be enforced through a constraint, which is fantastic for data consistency, but annoying if you misdesigned your data structure and have to change your foreign keys. But, don’t worry, it’s still possible to change this inside a single transaction with those four-ish steps:

Create the new key ID column, make it unique (e.g. serial)
Identify all tables which reference the (old) ID:
1.1. Note the table name
1.2. Find the referencing column
1.3. Determine the foreign key constraint name (see pg_constraint table)
For each referencing table:
2.1. Drop the existing constraint (from 1.3.)
2.2. Update the referencing key values to the new ones
2.3. Add new constraint for the new base table ID
Drop primary key constraint from base table
Create new primary key constraint on base table

Example

Let’s assume your application tracks users, which have external identifiers. At the moment you got a user table, which has this external ID as the primary key:

`user`

external_user_id (primary key)	user_name
12312	Alice
91823	Bob

Thus, all tables that reference any user have those long, external identifiers:

`user_permissions`

user_id	can_read	can_write
12312	true	true
91823	false	true

Now, if you want to gain flexbility and add an internal ID, you can start off with adding a serial column to users:

ALTER TABLE user ADD column id serial UNIQUE;

user would now look like this:

external_user_id (primary key)	user_name	id
12312	Alice	1
91823	Bob	2

Now, for every referencing table, you need to strip the constraint, update the values to the new serial ID and then add the new constraint:

ALTER TABLE user_permissions
DROP CONSTRAINT user_permissions_user_id_...id;
UPDATE user_permissions
SET user_id = (
SELECT id
FROM user
WHERE external_user_id = user_permissions.user_id
);
ALTER TABLE user_permissions
ADD CONSTRAINT user_permissions_user_id_...id
FOREIGN KEY (user_id)
REFERENCES user (id);

user_permissions looks now like this:

user_id	can_read	can_write
1	true	true
2	false	true

Finally, you can now change user.external_user_id without constraints, you just have to swap out the primary key in user:

ALTER TABLE user DROP CONSTRAINT user_pkey;
ALTER TABLE user ADD PRIMARY KEY (id);

At this moment, you can even put the external identifier into a different table, if you wish, thus possibly separating the concerns even better.

Bonus Content for Django Users

Swapping out primary keys is too difficult for the automatic Django migration assistant, so you’ll need to get your hands dirty. Obviously, you can do all those steps as raw SQL migrations, but then makemigrations will still nag you about changes in your model, thus you got to use SeparateDatabaseAndState, apply the changes with SQL and then tell Django what the effect is. For an example, see my code in osmcal.

Extra Bonus

If you’re working with raw SQL migrations, you might run into trouble and all you’ll get is some context-free SQL error message. For better debugging, you can use the sqlmigrate subcommand which will display the SQL queries instead of directly executing them. That way you can step through the statements and debug them step by step.

./manage.py sqlmigrate <app name> <migration name>

Making of the OpenStreetMap Calendar

Fri, 15 May 2020 12:28:17 +0000

Wikis have changed the landscape of information technology: No matter what organisation you‘re in and what topic you‘re working on, chances are that there is a Wiki, in which you can collaborate with others. Mostly without asking permission, you can put in notes, make lists and refine the material of others. Same with the OpenStreetMap wiki: Some pages are the standard source of information, like “How to map a…”, others are things relevant to just a single person, like pages where mappers list all the street they‘ve already mapped. And then there are things that have started with a good idea and gotten out of hand…

While collaborating on texts is good in most Wiki software, when it gets to tables and templates and structures, things get confusing, rather quickly. Sadly one of those examples is the Wiki calendar, which involves fiddling around with the custom templates, writing markup and someone rotating past events by hand. When we were talking about the OSM Wiki during events, people have been complaining repeatedly about the calendar. Some communities don‘t even announce their events in it, because they have shifted to Meetup or Facebook for better usability. There have been discussions how to solve it, maybe some sort of plugin or extension, but nothing has happened.

Inception

So, in early summer of 2019 I was mad as hell and couldn‘t take it anymore, so I sat down and started writing a standalone, special purpose web-based calendar for the OSM community. With maps!

Initially I wanted to utilize the infrastructure of the German OpenStreetMap Local Chapter, FOSSGIS, but they were busy generating rules about what I got to provide before getting a precious openstreetmap.de subdomain, so I went on my own and the domain osmcal.org was born.

The software is based on the incredibly powerful Django framework which helps with HTML templates, form handling, databases, authentication and much more. It‘s all open source, of course, as it should be. The issue tracker is on GitHub and already attracts a small community of people with bug reports, feature requests or general questions.

Progress

While osmcal started with a minimal feature set, development has continued and with the input from community members some features have shipped since then:

In October 2019, the possibility to “join“ events has been added, so organizers get a list of participants and as a participant you can see who else is going to come
In December 2019, iCalendar subscriptions have been made available
Since February 2020, event organizers can create surveys for participation, so they can ask people for e.g. their t-shirt sizes, their dietary preferences or skill level.

Special thanks to all reporters, especially @jbelien, @adrienandrem and @qeef for their collaboration and testing!

Future

Of course, at the moment, in-person meetings are out of the question, but this doesn’t mean that there are no events at all. Some are postponed, but many are happening online and those need to be scheduled and communicated as well. Thus, now it’s about time to integrate time zone support, which I initially deemed unnecessary as every event would have a location, but that has been disproven by the increased need of online events.

Also, I want more communities to integrate osmcal into their websites and wiki pages, so their lives get easier by not having to create and maintain their events on multiple platforms.

I was asked to consider applying for an OpenStreetMap Microgrant, so I submitted the aforementioned ideas with a request for funding. If you like to see those features and want to support further development, consider expressing your support on my grant application.

Making more connections

One new functionality that will be coming soon is community tagging, so organizers can create non-OSM-related events and have a dedicated feed with all their events. But more on that when it’s done. Stay tuned.

Faster Map Making With osmium

Sun, 22 Mar 2020 17:29:08 +0000

It’s sunday and I am fiddling around to make some maps, maybe to put them up my own wall and that’s the moment where I usually stuggle to get data out of OpenStreetMap into QGIS: Overpass is slow and won’t give you too much data, Shapefiles are ok, but this only works for limited regions and filtering and exporting OSMPBFs into GeoJSON is annoying.

Of course it would be better to have the data in a local PostgreSQL, but in order to get it there, you need to think about what data you want to load, but since you’re still iterating and fumbling with the data, this is not yet clear, so you’d need to have some cycle of loading, purging and reloading again. Annoying.

Then I remembered that you can use osmium to get all the data into PostgreSQL, without having to filter, which is not fast enough for tile servers, but easily fast enough for some experimentation.

Just get osmium tool and fire up:

osmium export -f pg -o taiwan.pg taiwan-latest.osm.pbf

Create a Postgres database and a table for the data:

CREATE DATABASE mapmaking;
\c mapmaking
CREATE EXTENSION postgis;
CREATE TABLE taiwan (
geom GEOMETRY,
tags JSONB
);

And now suck the data into the table:

\copy taiwan FROM 'taiwan.pg'

Now you got all the geometries in one column, the other holds all the tags. Now, you can fire up QGIS and use that table with some rudimentary querying action:

SELECT
geom, tags->>'ele' AS ele,
tags->>'name',
tags->>'name:en' AS name_en
FROM
taiwan
WHERE tags->>'natural'='peak';

Enjoy!

By the way, my work-in-progress greyscale map looks like this:

Towards Remotified Conferences

Sun, 15 Mar 2020 15:16:29 +0000

Our annual FOSSGIS conference just ended prematurely on Friday. Due to Corona we had to improvise and tweak a few things. Notably a lot of people couldn’t make it, either because they wanted to stay on the safe side for personal reasons or because their employers cancelled all business trips. This resulted in a lot of cancellations just before the conference, including a few speakers.

For those who still wanted to join, we wanted to have as much content as possible. We already live stream almost all the talks, except for a few who don’t wish to be televised, so remote consumption of the conference programme is nothing new to us. But because of speakers not being able to travel, we had to adapt and enabled remote speaking for the first time.

We are of the opinion that conferences are not just about talks, but about the connections we make, but considering the situation we can make more connections by allowing remote speaking than by cancelling the talks altogether. Unfortunately, we didn’t have much time before the conference to test thoroughly: The final decision to hold the conference just happened less than 48 hours before the opening.

Tech

Our conferences are usually held at universities, this year at University of Freiburg, which gave us the possibility to use the video conference calling system of the German National Research and Education Network, DFN, which is a web based application that luckily doesn’t need any addtional plugins.

In some cases we only had a few hours between knowing the speaker wouldn’t come to the conference and their scheduled talk. This meant we could only inform them about the technical necessities and ask them to join one of the test runs. Unfortunately not all of the participants were able to join our test, so in some cases we had a rocky start into the talk: In those cases the viewers had to wait for several minutes until the setup was settled. This is not perfect, but the pariticipants' expectations were reduced due to the circumstances.

Also, because there was not enough time, we couldn’t require a specific hardware setup. Thus some speakers had used either their notebook’s internal microphone, which even if not drowned by fan noise in most cases is too quiet, or they used Bluetooth headsets which had troubles with transmission, resulting in very distorted and unpleasant audio.

The video part worked much better: Most webcams are good enough for the picture-in-picture mode we used: Big slides on full screen and small camera window in the corner. Small video interruptions or low resolution are also not as annoying as audio hiccups.

Q&A

As I mentioned before, I believe that the exchange with participants and speakers is very important. Of course you can’t just talk in the hallway with the speaker after the talk, but at least for publicly suitable questions we gave everyone to get the microphone and ask the speaker directly in the Q&A session. Thanks to the N-1 audio setup the VOC has provided for us, the whole speaker-participant conversation worked quite well without any echo.

Speaker Experience

While the participants were mostly positive about the possibilities to watch the talks and being able to ask questions, some speakers noted that the presenting experience has not been optimal: They did not have a video feed from the hall, but rather just a view into the control room. I suggested to turn the camera into the crowd, but the VOC crew noted that for privacy reasons they don’t want to do that, even though that we asked for the participants' permission.

We muted the moderator’s microphone during the talk, so we the audience wouldn’t get any amplified noises, but this also meant that the speakers didn’t get any audible feedback from hall during the talk.

Developing the Concept

We didn’t really plan for all of this, but reacted to the situation, so not everything has been perfect. If we want or need remote speakers in the future, we need to make sure that a few things are in place:

Must Haves

Headset: Speakers need a wired headset or dedicated microphone, no excuses. Yes, it is easier to just talk into the notebook, but this does not work with a hall full of people. If your speaker doesn’t have suitable equipment, you’ll need to provide some. I’ve been told that people had good experiences with the Sennheiser PC 8 (Amazon), which is around 30 Euro/35 USD and features driver-less USB audio.

Internet Connectivity: Remote speaking is not for everyone from everywhere, you absolutely need a stable connection above all. Speaking from a home DSL or cable connection is good, but only if you don’t have packet loss and at least a few megabits of bandwidth. If speakers don’t have this at home or in the office, they definitely need to look for a better location. The location also needs to be tested beforehand, so there are no unpleasant surprises such as restrictive firewalls. Also: No wi-fi. Especially in metropolitan areas where multiple dozen wireless networks are competing, there will be unpleasant jitter. Speakers need to connect cables or, if sufficiently stable, use LTE, which has latency guarantees (no LTE over wi-fi though).

Light: Often more important than the camera itself is the lighting. Optimally natural light coming from the same direction as the camera or from the side. If that is not available, a properly illuminated ceiling works as well.

Test Run: Every talk needs to be tested. Optimally a connection from the venue should be established and tested for multiple minutes. Based on the experience, I strongly suggest to test a few days in advance, so speakers can acquire help or better equipment, if necessary.

Nice to Have

Chat Rooms: With FOSSGIS conference we are already streaming and recording all talks (most are published on the same day), so people who can’t come to watch, can do it from home. But it would be nice to facilitate the contact between speakers and viewers: It much easier to talk to someone spontaneously in the hallway than writing an email.

Conferencing Software: The DFN system worked quite alright, but it is a closed systems in which only research institutes can initiate rooms. In my experience Apple’s FaceTime is the most reliable video calling service, but it is only available on Apple devices thus not a good option for conferences. I would much more prefer some open source solution, that works for all operating systems and enables speakers and participants to create own rooms. I have been using Jitsi for some 1-to-1 video calls with good results. Probably it could also be used for conference setups.

Live (Audio/Video) Feedback: YouTube and Twitch have live chat for streamers. Especially speakers who don’t just read slides, but have a more conversational style of speaking need to get some live feedback, so audience video and audio could help as long as it doesn’t get too noisy.

Summary

We are not yet able to replace in-person conferences fully, especially considering that you can’t drink a beverage with the conference participants after a long day of talks, but considering the more available and affordable technology, we can at least save some of the tedious travel.

A big strength of conferences is that people are using a certain time frame to educate and to form new networks, so I doubt that fully remote conferences are going to take over soon, but extraordinary circumstances require extraordinary actions.

Why I Don't Believe in Boards Consisting of Volunteers

Sun, 08 Mar 2020 15:32:14 +0000

Currently, several people I know hold unpaid board positions in non-profits. Those organisations are tradionally run with slim budgets which suffice for the basic needs of the organisation. Often money is collected and spent for conferences, travel or project grants. But the budget is rarely spent for elected board members, which makes it very hard to fairly evaluate the performance of board members.

A common opinion is that those organisations do not want to encourage “professional politicians” that will try to get reelected because of the money.

But effectively no-one may critize the work of those volunteering board members: Someone did a bad job? Please don’t comment, because you don’t know how much time this member was able to spend. There weren’t any new initiatives? Well, you don’t know how busy they were!

And this makes me a bit uncomfortable: We’re part of NGOs or non-profits to support a cause, we hold elections for the best candidates, vote for someone and then some people get special titles. And if we think they did a good job, we’re going to re-elect them and reward them with the same responsibility and some more work.

This way of working excludes basically anyone with a full-time job. If you’re already working forty hour weeks and maybe have a commute of an hour per day? Well, it’s going to be hard to keep up. It could even result in more strain in boards: If you have individual board members which are able to work on board-related matters during work-time, you’ll make it harder for volunteers to keep up. And if you survive you may suffer burnout at the end of your term. This skews the representation as well: There is an overproportional number of self-employed europeans in boards.

In order to open up, I believe that organisations need to create plans to compensate for a minimum amount of working hours: It won’t be feasible to pay people full-time salaries, but they should get the opportinuty to get at least one paid weekday at median national salary.

Volunteers can still do projects, but a board position which has a constant stream of tasks? This is not a project, that’s a job!

Also, we should not drive away productive volunteers from their projects and convert them into policy makers, but we need to create a creative, resourceful environment for those who want to make plans reality. Otherwise our organisations will stagnate.

I think that organisations should strive to have enough budget to be able to compensate their administrators. I also believe that those organisations will be able to achieve more thus reaching more supporters, which again will make it easier to pay the boards. The creation of politicians can be limited by other means, e.g. the introduction of term limits.

Could We Create Binary Diffs for OpenStreetMap Planet Files?

Thu, 06 Feb 2020 11:29:08 +0000

Due to the recent spike in OpenStreetMap planet dump traffic, some people have wondered why so many users are redownloading the whole planet every week. The unfortunate truth is that applying replication diffs takes a long time¹ and is not 100% reliable.² I was wondering whether creating, distributing and applying binary deltas could help here.

The OSM PBF format consists of a sequence of blocks with data inside. Usually per block there is one kind of objects: nodes, ways or relations. Blocks can be up to 2³² bytes in size³, but usually capped at a much, much smaller size. The planet file currently consists of more than 14000 blocks, so every block is on average around 4 MB in size.⁴

If we assume that most changes are in fact additions to OSM, most changes would happen at the end of every section of the respective kinds of objects. This would mean that most blocks at the beginning would stay the same, as they rarely change and thus their delta would be zero.

The reality is quite different, unfortunately: Deleted objects move the position of all following objects inside the file, so changes cascade to every subsequent block:

It becomes very clear that with any change the block will look quite differently. Now, if we compare the planet from 2020-01-20 with 2020-01-27 there is zero blocks that stayed exactly the same. Of course this highly depends on the file packaging, e.g. when tested with an extract of Bremen⁵, 2020-02-01 and 2020-02-02 around 30 % of the blocks stayed the same.

The impact of shifting/cascading gets amplified by the used marshalling: In addition to the heavy use of delta encoded varints inside the file format, every block is compressed using zlib, so small changes can have an even more significant impact on the final block.

A final option is to compare the changes of every block in an uncompressed state: Let’s walk through two files and generate a blockwise binary diff.⁶

But even after decompression, the differences are too significant: The decompressed blocks are around 8.5 MB⁴, while the (binary, compressed) deltas between weekly versions are on average 2 MB. A blockwise weekly diff would be around half the size of the full planet, plus any new data at the end. Also, generating and applying binary diffs is far from being fast.

The diffability could be improved by keeping blocks as similar as possible, e.g. by removing deleted objects without allowing subsequent ones to move forward. But this would come at a cost of higher complexity during OSM PBF assembly. Furthermore deleted objects would not reduce file size.

Conclusion: With the current file format, there is no simple alternative approach and sticking to replication diffs is the only viable option at the moment.

Bonus Question: “Could we reduce file sizes by using a different compression algorithm?”

OSM PBF nowadays uses zlib compression which has a good compression ratio for data blocks.⁷ But how about the newer Zstandard compression, which promises smaller files than zlib at a much higher performance?

Bremen

	Compression Ratio	Decoding Duration (cumulated)
zlib	2.28x	464 ms
zstd	2.40x	169 ms

Planet⁸

	Compression Ratio	Decoding Duration (cumulated)
zlib	2.36x	1306 s
zstd	2.46x	383 s

This means that a 50 GiB planet file could be reduced by around 2 GiB while significantly improving read performance by using Zstandard. Such a change would not be a simple task: Zstandard compatibility would have to be implemented in a significant number of applications and libraries.

Whether such a change is worth the speed-up is questionable, since most of applications need to assemble geometries from OSM data, which is the biggest performance bottleneck in most usecases.

Around an hour on a quad core system with good internet connectivity and spinning hard disks. ↩︎
Diffs break sometimes, if for example very large changesets are uploaded and the processes that generate or apply the replication diffs exhaust their resources or run into timeouts. ↩︎
See fileformat.proto, message Blobheader. Please notice that this describes the zlib-compressed size. Average compression ratio is around 2x. ↩︎
On an OSM.org planet file, generated using planet-dump-ng ↩︎
Extract from Geofabrik, generated using osmium ↩︎
By the way, don’t try to run bsdiff on a planet file, the expected memory usage is around 900 GB. ↩︎
Fluctuates across the file (relations have a higher ratio than nodes), but averages around 2x. ↩︎
Tested on a relatively busy quad core Intel Core i7-2600 with spinning hard disks. Compression level for Zstandard was set to 20 (zstd supports up to 22), zlib compression level of OSM planets is set to 9 (maximum compression). Code from gosmparse has been used to perform the benchmark. No parallel decoding has been peformed. ↩︎

Synopsis of Tools for Importing OpenStreetMap Data into PostgreSQL

Mon, 11 Nov 2019 19:37:14 +0000

The de-facto standard for rendering OpenStreetMap data is PostgreSQL/PostGIS-based: The planet file or any extract is taken and then imported into the database. The same approach is useful for data exploration, connecting OSM data to own applications or doing data analysis. To get OSM data into Postgres, a suitable tool is necessary: Usually either osm2pgsql or imposm are used for this purpose.

While osm2pgsql is more mature and is used on e.g. the core OSM infrastructure, imposm is still not considered 1.0 despite being available for several years and proven in many installations. Then there is the swiss army knife of OpenStreetMap osmium, which a while ago received the functionality to emit data in a PostgreSQL-compatible format.

In this article we are going to compare the tools, show their strengths and weaknesses, have a look on resource consumption and the feasibility for different use cases.

Feature Set and Philosophy

The available tools have different approaches: osm2pgsql is a general purpose importer, with a focus on map rendering and geocoding. imposm is mostly used for map rendering, with the focus on generating tables that are pre-optimized for maps. osmium treats Postgres as an export format, without even connecting directly to the database, but just giving the option to pipe the data in Postgres' COPY format.

imposm allows customization through a mapping file, in which the user can describe which OSM elements will be considered, what tables will be created and how data shall be normalized (e.g. data types, sanitization).

osm2pgsql has two knobs that can be tweaked: Firstly, a style file that contains the columns that will be created and secondly, a Lua scripting API that allows for more complex transformations.

osmium usually just writes one table, with all tags in a jsonb column. The considered tags can be customized through white and black lists (exclude_tags and include_tags).

Many workflows use OpenStreetMap’s replication mechanism which allows to stay updated on the latest upstream data without having to download the full planet. OpenStreetMap provides minutely, hourly or daily diffs which can be ingested by compatible software. osm2pgsql and imposm support this natively: They can update an existing PostgreSQL database if run in diff mode (imposm) or slim mode (osm2pgsql). osmium can update on-disk OSMPBF files, but doesn’t provide a mechanism to replicate changes into a database.

Making the Right Choice

The choice of the importer software may depend on your external constraints: If you want to use a map style that includes osm2pgsql-compatible queries, it makes sense to stick with it, as the resulting database schemas are not equal and would need either additional views or transformations.

On the other hand, if you prefer the “one object type per table” nature of imposm, queries from e.g. osm2pgsql can be ported with some manual work. Also, imposm makes it extremely easy to normalize common OSM values, e.g. “yes” and “no” strings can be converted into native PostgreSQL booleans and implausible ones can be discarded at import time, thus reducing the need to sanitize values while querying.

For analytical tasks, osmium may be more useful as it won’t perform any tag transformations by default.

Software Strategy

Release, development, support and maintenance strategies are very different for those three tools. Especially for long-term system operation this might be a very important aspect to be considered.

osm2pgsql dates back to 2006 and has the largest number of contributors. Maintenance is performed by a community of people. Pull requests are processed in a timely manner, providing useful feedback. There are several releases per year. Packages are available for a wide range of operating systems/distributions: Debian, Ubuntu, FreeBSD, macOS (via homebrew) and others.

imposm started in 2012 and is under single-person corporate development and maintainership. Issues do not always receive feedback, pull requests are treated defensively and non-trivial changes are frowned upon. Releases are irregular, thus packaging is non-existent: Neither Debian, Ubuntu, FreeBSD nor macOS provide packages.² There is little development of new features.

osmium is a command-line tool based on the C++ library libosmium, which started in 2013, mostly developed by a single developer. Issues and feature requests are discussed and lead to development, if deemed useful. Pull requests receive feedback and are merged based on merit. Releases are frequent and release notes along with a consistent version numbering help communicate changes. Packages do exist in many distributions: Debian, Ubuntu, Fedora and macOS (homebrew) ship both library and the command-line interface, while FreeBSD only provides the library.

Resources for Import

Disk Space

Due to the large amounts of data it is generally advisable to work only with flash-based disks: SATA SSDs, or better: NVMe storage. Those have superior sequential write speeds and can retrieve random data much faster than spinning disks.

imposm will convert the data first into a more efficient on-disk format before flushing it into the database: By default it will create a cache directory which will take around twice the size of the original file. With the standard mapping the cluster size will be about 6x larger than the input file. Though, your milage may vary, as WAL configuration, toast settings and file system influence cluster size.

osmium holds an in-memory cache by default (more details in the next section). A full import needs around 9x of the original file size. No indexes will be created by default.

osm2pgsql needs around 7x the disk space of the original file using the standard mapping. The --slim option automatically removes intermediate tables after being finished with the initial import.

RAM

Due to the space-optimized design of the OSM PBF file format, for line or polygon assembly, traversing through at least one level of indirection is necessary. More specificly, all nodes are usually located at the very beginning of the file, followed by all ways. Thus, to assemble a line, the nodes at the start of the file are referenced. When selectively retrieving an object, jumping back to referenced elements is viable, but for unfiltered imports a full in-memory cache is a much faster option. Due to the extremely large amount of nodes, it is advisable to have lots of free RAM, optimally equal to the size of the input file.

osm2pgsql lets users specify the cache size and imposm utilizes an on-disk cache. osmium ships with several strategies and defaults to flex_mem, which will try to select the most efficient type. On memory-constraint systems, it can be run with dense_file_array, which will use a disk-based cache.

In the benchmark imposm utilized up to 3 GB of RAM (roughly equal to the extract size), osmium (with flex_mem) peaked at 5 GB and osm2pgsql has been configured to use up to 5 GB.

In addition to the importer’s memory usage, PostgreSQL needs buffer space as well. In the benchmark one quarter of the total system memory was reserved for its purposes. Due to the flexible memory management of PostgreSQL, memory can be over-provisioned for the time of import, as it will be reclaimed after the importers finish.

CPU

Nowadays, all of the three importers use multithreaded architectures, which allow to saturate several cores. Also, with the introduction of parallel workers for PostgreSQL, the database can work concurrently on the import operations.

The import duration will not scale linearly with core count, though. While it is advisable to have four cores instead of two and six will be slightly faster than four, there will be diminishing returns: With imposm import duration improved by 25 % when switching from 4 to 16 CPU cores. osmium meanwhile only improved by less than 10 %.

Import Speed

Because everyone loves meaningless synthetic benchmarks, there you go:

osmium imported the test file in 15 minutes.
imposm took 44 minutes to read, import and generate indexes.
osm2pgsql completed the import, including indexes in 62 minutes.

Querying

Query performance is mostly impacted by database size and indexes: A small database may be usable without indexes for simpler queries, a large database will slow down if scanned sequentially.

PostgreSQL brings a lot of (nearly) zero-cost abstractions (e.g. views) that help making data structures manageable, but good planning can improve data base size and thus performance.

Example: Table Structure

Let’s assume you have an imported database and now want to perform an analysis on shops. With an osmium-style single table import, you’d have a table with two columns: geometry and tags. When you perform a query, the database would need to scan all tags for the key shop.

SELECT tags
FROM osmdata
WHERE
ST_DWithin(
ST_SetSRID(ST_MakePoint(6.968, 50.943), 4326),
geom,
0.1
)
AND tags->'shop' IS NOT NULL;
-- Query planner:
-> Parallel Seq Scan on osmdata
(cost=0.00..1509477.60 rows=3148508 width=87)
(actual time=3958.424..7044.846 rows=52 loops=3)
[...]
Rows Removed by Filter: 7578245

A speedup can be achieved by an index on the geometry column, but in larger or densely mapped areas PostgreSQL would still need to scan many tuples.

CREATE INDEX ON osmdata USING gist (geom);
-- took 438 s
-- table data: 6047 MB
-- index size: 1546 MB

SELECT tags
FROM osmdata
WHERE
ST_DWithin(
ST_SetSRID(ST_MakePoint(6.968, 50.943), 4326),
geom,
0.1
)
AND tags->'shop' IS NOT NULL;
-- Now the spatial index will be used first and the
-- result set will be filtered down afterwards.
-- Query planner:
Index Cond: (geom && '[...]'::geometry)
Filter: (((tags -> 'shop'::text) IS NOT NULL) AND
('[...]'::geometry && st_expand(geom, '{...}'::double precision)) AND
_st_dwithin('[...]'::geometry, geom, '{...}'::double precision))
Rows Removed by Filter: 699511

To achieve a speedup, all elements with the shop key could be indexed³, but an index needs time to initialize, penalizes subsequent updates and inserts and takes disk space.

CREATE INDEX ON osmdata (( tags ->> 'shop'));
-- took 39 s
-- index size: 487 MB

Alternatively all shops could be written into a separate table during import, which could reduce the need for an index. But especially for exploratory development such requirements often cannot be planned upfront. If reimporting is not viable due to resource restrictions, a materialized view⁴ can be a good alternative as those are written to non-volatile memory.⁵

CREATE MATERIALIZED VIEW osmdata_shops AS
SELECT * FROM osmdata
WHERE tags->>'shop' IS NOT NULL;
-- took 18 s
-- materialized view size: 33 MB

SELECT tags
FROM osmdata_shops
WHERE
ST_DWithin(
ST_SetSRID(ST_MakePoint(6.968, 50.943), 4326),
geom,
0.1
);
-- Execution Time: 150 ms

Update or Full Re-import?

osm2pgsql and imposm allow for continuously updateable databases, but this has some limitations: A database cannot be updated, if is hasn’t been declared update-able from the start. Also they need additional on-disk storage. Additionally most users will experience bloat: The database size will only ever increase. Some techniques exist to reduce the impact of this, but an eventual reimport will be necessary, if disk space needs to be reclaimed.

Conclusion

If you want to process OpenStreetMap’s geographical data in PostgreSQL you are presented with some good options:

osm2pgsql is the solid choice, with wide compatibility and a long project history. It’s resource efficient and offers some customization options. It also handles diffs.

imposm might not be very well supported, but offers quick options for preprocessing and is disk-efficient, if needed.

osmium might be the newest, but since it’s a thin layer on top of the legendarily efficient libosmium, it’s the quickest here. It has the fewest customization options, but perfect for everyone who wants to do their work mostly in SQL.

Annex

Benchmarks have been performed with a Germany extract (3.0 GB), on a virtualized server with four dedicated cores, 16 GB RAM and NVMe storage, formatted with ext4. The OS is Ubuntu 19.04.
- PostgreSQL configuration values:
  - shared_buffers = 4GB
  - temp_buffers = 32MB
  - work_mem = 1GB
  - maintenance_work_mem = 64MB
  - autovacuum = off
- Every run has been performed with a freshly recreated cluster and a PostgreSQL restart.
The 16 core benchmarks were performed in the same setup, except for RAM, which increased to 64 GB.
The shop query example has been done on the NRW, Germany extract (635 MB).

Consulting is available by the author.

Banner picture by Fallaner, licenced under CC BY SA 4.0 ↩︎
Some of those do have packages for imposm 2, which is the python-based predecessor of imposm (3), which is fundamentally incompatible and mostly unsuitable for the tasks discussed in this article. ↩︎
See https://www.postgresql.org/docs/12/datatype-json.html#JSON-INDEXING ↩︎
See https://www.postgresql.org/docs/current/sql-creatematerializedview.html ↩︎
Materialized views and a continuously updated database can be tricky to handle as those are only updated on demand. ↩︎

Points of Interest as a Service

Mon, 15 Apr 2019 16:05:34 +0000

When building a location based service, access to public geo data like points of interest (POI) can be cumbersome and expensive. There are several proprietary databases which have specialized on access to POI data, but with the ever growing collection of data by OpenStreetMap and the collaborative power and its mature ecosystem of professionals and volunteers OSM is now a leading data source in many areas.

OpenStreetMap now tracks over a million restaurants and bars, 310,000 hotels, nearly 150,000 view points, 140,000 hospitals and over three million shops worldwide. They are not only precisely geolocated, but often also cover additional data like opening hours, telephone numbers or wheelchair accessibility.

The data and a lot of the technology in the OSM ecosystem is open, but for those who want to build upon it and use its power without worrying about setup and hosting, the possibilities are often limited: raw OSM data needs several processing steps and scaling up to a full planet access can be prohibitively expensive.

For this purpose, today we are launching OORA, a cloud-native API service that allows you use OpenStreetMap data without having to worry, so you can focus on your application.

Technical Details

To get all nearby convenience shops, just get

https://api.skowron.eu/oora/v1/shop/convenience?
center_lat=50.95137&
center_lon=6.95697

which will return something like this:

[
{
"type": "Feature",
"properties": {
"name": "Maxi Markt",
"shop": "convenience"
},
"geometry": {
"type": "Point",
"coordinates": [6.95652930, 50.94593490]
}
},
{
"type": "Feature",
"properties": {
"addr:city": "Köln",
"addr:country": "DE",
"addr:housenumber": "12-14",
"addr:postcode": "50668",
"addr:street": "Riehler Straße",
"brand": "REWE",
"contact:phone": "0221 9726040",
"contact:website": "https://togo.rewe.de/",
"name": "REWE to go",
"opening_hours": "24/7",
"shop": "convenience",
"wheelchair": "yes"
},
"geometry": {
"type": "Point",
"coordinates": [6.96184250, 50.95249260]
}
}
]

The data is licensed under the ODbL, Copyright by OpenStreetMap contributors. See X-Copyright HTTP header.

Getting Started

You can sign-up for the technology preview by entering your email address here:

You will automatically receive an API key, which will allow you to use the service for free. The free evaluation will last until at least May 31, 2019. You will be automatically notified before the production-grade product launch.

Please provide the received API key inside the Authorization HTTP header.

If you want to perform more than 100,000 requests, please contact us. Please also let us know if you want to use the API in production.

There is no warranty for the free service. If you need professional support, a custom deployment or consulting, please contact us first: oora@skowron.eu

Give Spaten a Spin: Introducing a Data Repository

Thu, 21 Feb 2019 11:03:23 +0000

In 2017 the Grandine project has been introduced, which helps working with geodata in a streamable fashion. At the core of the project the new file format “Spaten” for passing around geo data has been introduced. A lot of people reached out to me, wanting to explore the possibilites.

In order to facilitate work with the provided tooling, the Spaten file repository is released today: It is a collection of Geo Data files which are free to use and can be processed with any Spaten-compatible tool, like the Go library, the Python implementation or the Grandine Tools.

For now, the datasets are all based on Natural Earth and therefore released into the public domain. The data sets contain physical and cultural vector data at different scales: 10 m, 50 m and 100 m resolution.

The download overview is reachable under https://thomas.skowron.eu/spaten/download/

The Spaten file format is very efficient and therefore optimal for geo data applications at any scale: Decoding is at average 5x faster than GeoJSON and file sizes are up to 50 % smaller.

To generate a vector tile set from a spaten file, just type:

grandine-tiler -in ne_10m_airports.spaten -zoom 10

Envisioning a New Approach to Geodata Processing

Tue, 16 Oct 2018 11:05:00 +0200

This is a recap of my work and the subsequently gained learnings of the last two years. Parts of it have been discussed in my talks on FOSSGIS and SOTM conferences, previous blog posts or podcasts.

Today

Processing geo data can be approached in two ways: Either the algorithm is already clear and the process can be automated and data can be processed directly, or: a human element is needed for processing in some kind of exploratory phase.

There are processes which are somewhere in between, because they are generally application driven, but need human interaction in some step, e.g. generalization, in which a human preprocesses geometries, but can later pass those into an automated pipeline which will use those for e.g. low resolution purposes.¹

Manual processing is especially present in the traditional geospatial work: Surveying, spatial analysis and print maps. This is most often done using desktop GIS applications, like QGIS. For persistence either shapefiles or a local PostgreSQL/PostGIS database are used frequently.

Automated processes very often have PostGIS at its core, too. Projects like OpenMapTiles even use a Postgres database to import all data first just to retrieve all data from it after that.

For manual processes computational performance is seemingly not very important. Operations controlled by humans are often the limiting factor. Nonetheless in larger data sets it can become tedious when loading pauses become noticeable: Speaking from past personal experience, waiting for Tilemill to rerender the current view after a minor change can be very stressful and slow down any iterative development.

Automated processes often have very different characteristics, because they generally involve much larger data sets and mostly require a sequential processing of all present data. Typically those are workloads which are not suitable for a full relational database system.

Graphical map making is, as mentioned before, a mixture: Often large data set, but with a manual element. Especially in early stages, when a complete style needs to be designed, a lot of iterations may be needed, so low latency is necessary. Later on, throughput is much more important, which probably is the reason for the increasingly large popularity of vector tiles, because it offers both benefits: Low barrier for style changes while being very easy to scale.

Spatial awareness is less important than it might seem for a lot of applications: Layers or other categorizations of features need to be considered, too, which often lead to a full read and write cycle: Every feature needs to be retrieved and read to be written. For such workflows, ACID semantics which are present in database suites are not needed.

Also, big data sets are coming along: They are large enough to overflow available memory, but generally still small enough to be persisted on a single disk/machine.

Sharing data sets can often be a large pain point: If sharing a central PostGIS is not feasible (e.g. because of too high latency or firewall and policy restrictions), passing on data can hinder collaboration: Shapefiles are severely limited, spatialite is too slow for large-ish data sets and shipping off a complete PostgreSQL cluster can be quite difficult, especially for non-programming users.

On the other side of the fence

Aside from classical geo spatial work (like in government agencies) and neo-geographers (like OpenStreetMap contributors) a third class of users with geospatial needs have emerged: Companies which develop products like location based services or social networks, which are not in the tradition of using GIS tooling. They use software that is suitable to their existing toolkit, programming environment and scale. They generally do not participate in standardization, but rather stay in their own ecosystem.

While generally geospatial data sets are comparatively small in contrast to user data, some organizations prefer to shard it across nodes and make additional steps to make data more available.

Trends

Using tiles is not a new concept²: They have been introduced when web-based maps have been brought to the public. There is no need to download the whole data set, just to have a look at a geographical context of a few street blocks.

With the inception of vector tiles, the same mechanism of reducing data delivery to small spatially index files has been adapted into a wide variety of applications: The obvious map rendering in browsers³, but also routing on tiles⁴ and even geocoding⁵. Generally, engineers in VC funded companies seem to enjoy to build distributed toolchains in order to generate vector tiles.

Also, more and more applications have an initial import stage, in which they transform all available data using domain specific application logic to be able to operate.⁶

In-Process Recap

Automated geo data processing suffers from performance bottlenecks: Especially during development this can be tedious. Databases have a lot of nice characteristics, of which a lot is not needed for geospatial work.
Distributed processes have huge performance impact (read: dozens of instances working on days for a planet-sized data set)⁷
Not everyone is able to process worldwide/planet OpenStreetMap data on their own machine (OSM full planet needs >40 GB RAM).
- The power of OSM comes from grassroot approach, not from large comities!

How to Make Everyone Happy

It becomes clear that there is a need for a resource efficient approach, which solves common problems. Computers have become very fast for some classes of problems while other things are still rather slow. Generally speaking sequential processing of similar data has become very fast: with heavily-pipelined CPU cache architectures, the spread of GPU based processing or very efficient JIT systems. If embraced, we could benefit from huge performance gains by eliminating expensive index traversals and memory seeking.

It needs to be possible that users can use data sets in their desktop GIS software while being able to write scripts or programs that can process those very efficiently – optimally in the programming language of their choice, while being able to interact with other software.

By the help of good libraries in diverse programming environments it is possible to construct an approach of stream in, stream out: Every written application takes a stream of spatial features, processes each one by one (or optionally multi-threaded using workers) and returns the output as a stream. Thus an application needs only a fraction of the memory that a full data set would require. From there on, only the speed of the CPU, the number of cores and the complexity of the applied algorithm would limit the performance.

Use Cases

Let’s have a look on different use cases (in a simplified way) and judge how they could fit into such a mode of operation:

Multilingual maps (transliteration/transcription)
- Work steps:
  - Traverse every feature
  - Look at location
  - Look at name tags
  - Return every feature with appropriately transformed name
Tiled maps (e.g. vector tiles)
- Work steps:
  - Traverse every feature
  - Look at location
  - Determine affected tile
  - Append geometries to appropriate tiles
Data analysis
- Work steps are very dependent on the use case, but data sets are mostly not that large and can be scanned more efficiently than indexed and retrieved on every operation. Especially volatile data has a bad index utilization.
- When multiple people are involved, a shared DB can make more sense
- Also: PostGIS offers a lot of intuitive tools for processing
Geocoding
- Geocoding software like nominatim has special importers, which have to process every feature on initialization, but mostly do not rely on a spatial database.
- Scanning every feature and applying an algorithm is already exercised.

For multilingual maps, tiled maps or geocoding the streamed approach is pretty natural and could be applied in an efficient manner. For data analysis, it depends on the task and the people involved. Especially in exploratory phases, a database that allows concurrent access might be useful.

Limitations and Obstacles

Of course adoption needs aside from strong benefits also a reduced set of obstacles, but obviously not everyone is incentivized or motivated to switch to a new paradigm which is a hindrance on its own. Aside that non-programmers still want to use their GUI software without much change. When loading times improve or the friction of exchanging files gets reduced users might notice, but missing features or changes in the daily workflow will be much more painful. After all, almost no-one complains about a coffee break while the computer is busy.⁸

PostGIS power users will probably never switch: They are used to a very specific declarative mind set, which is so wildly different that friction might always outweigh the speed or convenience benefits. Also, enterprise users will probably not consider switching before everything is standardized.

When specifically considering an implementation of this approach and looking at the most probable reason why most software uses PostGIS as a geospatial toolkit, it seems that the majority of spatial libraries are of poor quality and correct handling of complex geometries is tricky, even with the support of specialized packages.

Opportunities

This novel approach should not only cover existing use cases, but also bring new opportunities to the table: Hobbyists should be enabled to process worldwide data sets on their local computers (or on small, cheap cloud instances, if you prefer). Furthermore applications could start being decoupled from a specific database implementation, e.g. CartoCSS projects contain inline SQL queries⁹, which makes them not only not portable to other databases, but also generates a significant effort when trying to port it to a different table schema.

The Suggestion

To recap, we should treat geo data as a stream of features that can be passed from application to application in an efficient manner. The same format to pass features should also be used for persistence on disk or sharing using the cloud. To pass data between applications, pipes can be used, reducing disk footprint and enabling more agile processing.

This train of though is encapsulated into the Spaten file format, which I suggest as a basis for pipeline processing. This format is already used in the Grandine tools.

State of the Vision

A foundation has been laid: The format for serializing features has been defined and a first library implementation in Go is available for integration. This is used in the Grandine toolset in order to pass streams in and out, where applicable:

grandine-tiler can build vector tile sets from streams
grandine-spatialize converts OpenStreetMap into a spatial format, serializing it as a Spaten stream
grandine-converter reads geojson and can stream out Spaten data

In discussions at conferences, meetups and hack days there is a general interest in the technology and some people are willing to have a deeper look or integrate Spaten into their software in the future.

How can you help?

Of course this is just a first step into the journey. I think a lot of applications and people can benefit from a faster, more efficient toolchain. I strongly believe, we can make open applications more cooperative and make tools written in different environments and languages more useful to everyone.

But for that I need your help: Tell me, what you are up to. Write an email, talk to me at some event. You can also help test the existing libraries and command line tools. If you find bugs, write an email or file an issue. Also fuzzing or expanding benchmarks and tests would be great. Go experts can help me write a better polygon clipping function.

I am also looking for people willing to help out build serialization libraries for more programming languages (especially JavaScript/node.js and C). If you are interested, let’s work together!

Also if you want your company be part of this, you can make me work for you as a freelancer.

Acknowledgments

Special thanks to the Prototype Fund, the Open Knowledge Foundation Germany and German Ministry of Education and Research, who supported the project and funded six months of development.

Also all of this would not have been possible without Jochen Topf, who convinced me not to jump into the Postgres rabbit hole again and supported a lot of initial thought.

In the end I wouldn’t have come so far if not for the great conversations I had with people from the OpenStreetMap universe: If you have ever written an email, talked to me or just listened to my insane ideas: Thank you, you have been a great motivator!

OpenStreetMapData provides generalized coastlines for use in low zoom levels. ↩︎
OSGeo already published TMS in 2006 ↩︎
Mapbox GL JS ↩︎
valhalla ↩︎
carmen ↩︎
OSRM needs to extract the road network from source files before being able to run ↩︎
Approaches like OpenMapTiles need days to process a full OpenStreetMap planet. ↩︎
Relevant xkcd ↩︎
Example from OpenStreetMap CartoCSS code repository ↩︎

Orphaned Gunicorn Processes with Supervisor

Mon, 02 Jul 2018 23:06:00 +0200

Python web applications generally expose a WSGI interface for client communication, which is rather unusual if you are used to more novel approaches like those of Go applications, where APIs are generally served via HTTP. Most applications have an HTTP gateway in front, either for load balancing, high availability or TLS termination, so a Python application, like e.g. Django projects need an HTTP server which interfaces with WSGI.

WSGI servers also deal with worker spawning, because of Python’s Global Interpreter Lock, which would otherwise limit Python applications to be executed on one thread at any given moment.

A popular WSGI server is Gunicorn, which itself is written in Python, so integrating it into an application is just one install command away. Launching a web application just takes a /app/venv/bin/gunicorn myapp.wsgi. By specifying a number of workers using the --workers parameter a higher parallelism is configurable.

Now, in order to ensure that a web application is started on boot and restarted if it fails for some reason, you need an init system. A well-regarded approach is to use supervisord, which makes it easy to write unit files, which define how an application will be run:

[program:myapp]
environment = DJANGO_SETTINGS_MODULE="myapp.settings"
user = myapp
command = /app/venv/bin/gunicorn myapp.wsgi

A handy feature of Gunicorn are graceful restarts: You only need to send a HUP signal to the master process, which will finish handling all current requests while rolling over to the new version of the application. Though, if you change supervisord’s config, you may need to truly restart the units.

This can lead to an unexpected behavior: The Gunicorn master process gets seemingly shut down, but the spawned children processes are still there. When supervisord tries to launch the new master, it gets stuck as the binded port is still occupied. A look on htop reveals the problem:

kernel
`- /sbin/init
`- /app/venv/bin/gunicorn myapp.wsgi --workers 2
`- /app/venv/bin/gunicorn myapp.wsgi --workers 2
`- ...
`- /usr/local/bin/supervisord

Normally, the Gunicorn application server processes should be children of supervisord. But instead they have been orphaned and are now children of PID 1, the init system. They can’t handle any traffic, because the master is already gone, thus they are effectively deadlocked.

A temporary solution is to kill -9 the orphaned processes, which will allow to respawn the application as soon as the supervisor retries the next time. This will unlock the application for the moment. Still, with the next supervisor restart the problem will appear again.

The real cause of this problem is a specific supervisor behavior, which will send signals only to the controlled process but not cascade it to its children. This can be resolved by setting stopasgroup=true in the unit file. Setting it will prevent orphaning.

Note that restarting/reloading supervisor is not needed in most cases, e.g. when you are rolling out a new application version, it suffices to use supervisorctl signal hup myapp, which will prevent a service disruption.

The recommended supervisor unit configuration for a Python application run with Gunicorn is:

[program:myapp]
environment = DJANGO_SETTINGS_MODULE="myapp.settings"
user = myapp
command = /app/venv/bin/gunicorn myapp.wsgi
redirect_stderr = true
stopasgroup = true

Open Data: Best Practices

Fri, 02 Feb 2018 18:48:00 +0100

In case you are able to read German and are interested in pursuing an Open Data strategy, you might want to look at my Guide to Open Data.

Grandine: Vector Tiles, Summary July 2017

Tue, 01 Aug 2017 00:52:00 +0200

Continuation of the documentation of my Prototype Fund related work. Previous summaries: March, April, May, June

This month I concentrated on picking up some loose ends and refining the both command line tools spatialize and tiler.

The other hard thing in programming

I’ve been reminded that there are already tools that are called “tiler”. I don’t intend “tiler” to be a canonical name for this tool, but as it isn’t production ready yet I didn’t want to invest too much time into a good name. Nevertheless I consider “tiler” and “spatialize” to be subcommands of grandine. So it actually might be called grandine-tiler in the future. But this is not set.

Spatialize

Spatialize is the tool for converting OSM data into a geospatial data format with proper geometries (collects ways and relations and assembles polygons, if specified) and feature and layer name mapping. For starters I implemented parts of the OpenMapTiles Vector Tile Schema.

Tiler

Tiler has been rather slow if it needed to cover large areas, because it has been seeking in potentially large lists. With newly introduces changes it builds a rtree for fast geometry lookup if a lot of geometries are present and/or a lot of tiles need to be generated. Furthermore the work is now spread across a number workers, so it is able to utilize all CPU cores.

`lib/spatial` changes

The internal spatial library has been refined, but also received a new feature: It is now able to simplify lines using the Douglas-Peucker algorithm.

Obstacles

A thing that gets increasingly frustrating is the polygon clipping algorithm which still does not work correctly in some cases. If you have any insight into this and maybe have already implemented Weiler-Atherton I would love to hear from you.

Time ticking

There is one month of funding left for the project, so I am trying to push out a few more cool things, because on August 31st it’s demo time!