Thomas Skowron's Blog

Why the Cloud Isn't Getting More Expensive (Yet)

Fri, 27 Feb 2026 16:03:14 +0100

When Hetzner, a popular German hosting provider, recently announced price hikes of more than 30% for their services¹, this did not go unnoticed. The reasons are pretty clear: Hardware component prices jumped massively. When I accidentally bought the wrong RAM sticks in November and I needed to return them, in a two-week span the price doubled. I imagine the shop must’ve felt quite happy with my mistake – although I doubt that shops really have feelings.

Germany is not the first place I would start a hosting provider: Electricity is expensive, finding enough land for data centers is difficult, permitting is always a struggle, and personnel costs aren’t that cheap, but through fierce competition in the early 2000s a scene of reliable, low-cost providers emerged. The shrewd ones that managed to work within these thin margins have stuck around.

When AWS started building their cloud offering, their pricing structure wasn’t beholden to the same logic. SmugMug, their first enterprise customer, was happy to switch to S3 without saving any money. Passing off the responsibility to a third party at cost parity made sense to them. Of course this was 2006: All hardware was still getting dramatically cheaper. Between 2006 and 2018 hard disks dropped in price more than 90%.² Not even the floods in Thailand in 2011 could meaningfully disrupt this trend.³

In the early days, AWS passed some of these savings on, possibly to deter competitors from realizing that the cloud was a good business to be in. That deterrence, by the way, seems to have worked: All other successful entrants emerged from their own needs for cloud services. The price cuts stopped around 2014 and from then on, prices have found stable ground: Their mindshare was established, other big name providers had similar pricing, and customers wouldn’t be switching en masse to smaller providers.

AWS did launch some new products that allowed a cheaper on-ramp, like Lambda, which allowed to serve code without paying a flat rate for the substrate it’s running on, along with other “serverless” services that often start out at 0 USD, but have a sharp cost curve once used heavily.

With their AWS Graviton architecture, they also presented some cheaper EC2 options: A t4g.nano is just around 3 USD/month. Also, it only has ½ GB of memory, which makes it unusable for anything that isn’t engineered specifically to run within these confines.

The sharply rising hardware prices in recent months are attributed to the AI boom, or bubble, as some prefer. There is some circumstantial evidence: AI companies are raising insane capital, which they use to buy GPUs from a limited manufacturing pool. GPUs need memory chips, therefore GPUs and memory are getting expensive. Everybody who is not an AI unicorn needs to buy these with their real, hard earned cash, which may seem massively unfair.

But unless you plan to build a gaming rig, this is just whinging, because chances are, you have already been paying a premium. Apple is charging 400 USD for a 16 GB RAM upgrade (to 32 GB). And if you are like me and don’t find this completely outrageous yet, AWS charges for a mere 7.5 GB memory upgrade (between t4g.nano and t4g.large) over 550 USD per year!

Of course, buying instead of renting gives you the magic of depreciation: In an earnings call, AWS has reported that they run hardware for five years, meaning that even at current memory prices they have a margin of more than 15x.⁴ With a 3 year reserved instance, it shrinks into a pitiful 6x. Assuming AWS is paying retail price, which I am sure they have figured out a way not to.

For all the thin-margin hosting companies, this bears an existential risk: Their customer-facing price per GB of RAM often hovers around 10 EUR (~12 USD) per year, which only starts to break even after more than 2.5 years. Assuming the memory prices stay flat.

The market does not show any signs of relief: The number of memory manufacturers has shrunk to just three, and new contenders will find it difficult. DDR5 memory has extremely tight timing tolerances, and the big ones are utilizing EUV lithography to manufacture those chips. CXMT, a Chinese manufacturer of memory, has started a DDR5 line, but as Chinese entities are barred from importing current-generation lithography equipment, they are still struggling with engineering challenges.⁵

Chinese labs are working on their own EUV technology⁶, which may reach maturity in 2030, but everybody would prefer to be Nvidia instead of just another memory manufacturer, so it’s not clear how new capacities would be utilized. Cutting edge hardware takes years or decades to get into production while predictions are imprecise.

This gold rush sparks a lot of interest in competition for Nvidia, as nobody likes to pay 25k USD for a single H100: Google invested early into TPUs, and Cerebras offers powerful hardware for training and inference you can buy. Still, any of them need to utilize manufacturing capacity which is massively constrained and sold to the highest bidder. Fat margins insulate from icy market conditions, but attract hungry competition.

Even if AI company valuations don’t stay as high, LLMs are here to stay and will continue to devour a lot of the available hardware resources. But if capital dries up, purchase orders may disappear overnight, triggering a sell-off. OpenAI already slashed their spending plans by over 800 billion USD.⁷

The three big hyperscalers are growing massively in revenue not least because of the huge spend by AI companies, so they don’t have to worry about decreased margins in other areas – at least for the moment. The big question will be if the revenue streams from AI services will be sufficient, once the products mature.

In the tech industry we have been massively spoiled that costs for hardware only ever went down, and somebody else had to do all the hard work to get there. Sloppy software engineering could just gobble up all the resources. But what happens if the free ride is over?

The cloud revolution created generations of software developers who do not know how to run on hardware, so they have to rent it from others. The AI revolution may cause developers to not know how to build software, thus having to rent it by the token. The question isn’t really whether the cloud will get more expensive: It’s how many layers of rent you will be willing to pay.

https://www.hetzner.com/pressroom/statement-price-adjustment/ ↩︎
https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ and https://mkomo.com/cost-per-gigabyte ↩︎
https://www.bbc.com/news/technology-15534614 ↩︎
Assuming a current market price of 25 USD per GB of RAM and a five year depreciation schedule, while charging roughly 75 USD per GB surcharge per year. ↩︎
https://www.tomshardware.com/pc-components/dram/chinas-cxmt-reportedly-delays-mass-production-of-ddr5-chips-to-late-2025-state-backed-manufacturer-could-still-be-disruptive-market-force ↩︎
https://www.reuters.com/world/china/how-china-built-its-manhattan-project-rival-west-ai-chips-2025-12-17/ ↩︎
https://www.cnbc.com/2026/02/20/openai-resets-spend-expectations-targets-around-600-billion-by-2030.html ↩︎

Licenses: Trust and Promises

Fri, 04 Jul 2025 14:02:11 +0200

Fundamentally there are two types of perspectives inside Open Source: true believers and opportunists. Also those who still secretly think it’s a disease – but let’s leave those aside for the moment.

I have wandered in the “true believer” FOSS fundamentalist circles for a long time, and it never felt like a practical ideology to me, so I’d consider myself an opportunist, despite having written Open Source software (OSS) for free and for money, having contributed to codebases of others and donated my own money to it and made others direct their cash.

Either way I think that Open Source is a huge chance, irrespective of whether you are building it for yourself, for your company or just harvesting stuff made by others. And it does not have to be extremist either: Even if it’s not really Open Source, meaning under an OSI license, published openly, but even if it’s software that the rest of your organisation can access and send pull requests to, or if it’s code that you send to your customer after the invoice got paid. These small things can have an immense effect on the impact that your developers can have and the trust you build in and outside of your organisation.

Even if your customers might not understand the value that you are sharing the source code with them, it might push you to produce better artifacts.

At its core Open Source might be seen as a legal contract: You are getting a piece of code and technically speaking you can run it as long as you want. Practically speaking you will want to receive updates at some point, either to get new features or fix security problems. The very most sophisticated consumers of software may become contributors themselves: It may be more practical to develop patches in-house than to contract either the original author or switch to a different software.

Since standard licenses are often quite simplistic – grant permissions and exclude warranty – the social contract is crucial. And all the projects that are altering the legal contracts unilaterally are putting a strain on the social aspect.

Future expectations are central to that deal, even if they are not codified: Ten years ago when adoption of Open Source software packages was debated inside of companies, doubts about the longevity were a major contention point. This aspect has become less important, because Open Source as a whole got more accepted and stable. License choice and compatibility are more central aspects of due diligence these days.

With the ongoing license change wave (e.g. Redis, CockroachDB, Terraform) the function and reality of OSS must be questioned: Why are companies deciding to publish their source code? And why do customers want Open Source?

Universal Toolmaking

Tool making is a profitable business, for producers and consumers alike. People are proud of the tools they own: sometimes because a cheap tool is making one unreasonably productive or because they have bought something of extraordinary quality that enriches their enjoyment in the process. ASAKA, a Japanese tool maker, exists since 1661, primarily manufacturing spades and shovels. Their business model works for over 350 years despite not having an end-user agreement that dictates that you may not use their tools in for-profit endeavors.

Derek Collison, founder of NATS, bemoaned in an episode of Changelog¹ that some mega corporations have policies that block them from paying at least for certain kind of Open Source tooling. This framing makes the complaint look extremely reasonable: A huge player having an almost comically bad policy that takes the fruits of a much smaller entity and not paying anything.

From the consuming companies’ perspective they seem to be legally in the clear: The Open Source license allows them to use it as they see fit; they can even make private changes to the code since it’s licensed under Apache 2 license. Why should they pay for something they are already allowed to do? Sure, they didn’t pay for the shovel, because it’s been gifted to them.

Customers of software businesses are often subject to absurd licensing rules: Limitations on concurrent usage, agreement to invasive audits or telemetry, and chiefly “no warranty”, aka the software vendor not guaranteeing that the product performs any function. Open Source corrects this relationship somewhat: The customer can terminate the business relationship if they are unhappy. Even if you have bought a perpetual license, only some form of OSS give you the ability to change the software after the fact.

Upstreaming changes comes with its own troubles: Some vendors are not thrilled about (potential) customers just submitting patches. Many of them would rather make a service contract sale and develop the patch in-house from scratch, instead of inheriting a maintenance burden. But by rejecting these contributions upstreams are risking the customer creating a fork. Some vendors are not integrating the customers’ patches but instead engineering their own solutions that fulfill the requirement. From the outside it looks like the original authors are the only contributors, but in the long run it is a unsustainable business practice if the customers are not paying.

More integrated solutions are creating more value and are often able to achieve higher margins: A shovel manufacturer will have a slimmer margin than a construction company. A scrum ensues when the lower value producer (creator of Open Source Software) cannot even collect their small margin, because the upstack vendor (a public cloud company) is paying zero while selling something more useful.

Cloud providers or even hyperconverged infrastructure vendors have little problems collecting money from the value they provide, mainly because they have gauged correctly how much of the value they are allowed to capture. Although they are at risk of these dynamics shifting, as I alluded to in my Go Big or go On-Prem piece. Some Open Source Software companies are able to earn good money, but measured against the value they are creating, the capture ratio is rather low.

From the perspective of support contracts I find the “Business Software License” (BUSL) by default, Open Source after two years construct not violently unreasonable: Many software and infrastructure contracts are negociated for a multi-year period, and after that expires they have an escape hatch: Whatever they originally bought will become Open Source after the contract ends. If the vendor turns out to be a dick or gets sold to one, you have a kind of insurance and can keep using what you’ve already got.

Execution matters

From a customer’s perspective some OSS vendors are making massive mistakes in their execution and later blaming greedy non-payers for their ills: I still find it baffling how disappointing support and sales sometimes can be.

Open Source allows for a low-friction start: You download a package and start going. When you have to buy software, even if you have access to a trial version (which often isn’t even long enough to cover the sales process), it slows any adopter down by weeks or months.

A customer of mine has been using the Open Source version of a database software. When we hit some issues, we tried to find a solution through support, which wasn’t able to assist us in any meaningful way. Their paid cloud offering was an underdocumented mess and wasn’t even competitive against running the open version on-prem ourselves. The sales process was slow and didn’t resolve any of our questions. In their newest version they finally decided to cripple the Open Source version and shifted focus to their enterprise offering. OSS hasn’t failed here: It’s been their strategy and execution.

When companies close down their source it is easy to be cynical and claim a money grab or see their open efforts as a way to lure customers into their captivity. It definitely tarnishes the reputation of these companies, because it will be much harder to believe them that they will not alter the deal even further. The customers, especially these of VMWare or Oracle, already know this asymmetry of power. And the aftermath of the Terraform agreement change and the emergence of a decentralized OpenTofu shows that this kind of rug-pull may have unintended consequences.

Companies like Tailscale are showing how it’s done: Their product is building upon Wireguard (an Open Source VPN software), and an eager technologist might argue that you could just use that. But their execution makes it so easy to pick them instead: You can create an account using self-service, much of the software they are offering is Open Source, you can even use a different backend. If you want to go further, their support staff can actually help and their sales is proactive. They are pushing themselves to provide more value to their customers, so they don’t have to lock you in.

If you are a “pure play” software company, launching your product as Open Source and hoping to make it a sustainable business through e.g. enterprise support contracts, I do still think that it’s possible, but your fundamentals need to be absolutely solid: Your sales staff needs to be aware what they are up against, your support needs to know how to effectively help. Your strategy must be somewhat transparent or consistent. Your investors need to know that monetisation strategies may not be straight forward. Don’t trade a boost in quarterly numbers for reputation.

If you are on the other side and are enjoying the benefits of Open Source, think about where you already spend your money: Are you maybe paying for a cloud account that doesn’t have good support and doesn’t finance the underlying OSS? Maybe paying for the hosted solution offered by the producer of the software makes more sense? If you are spending big bucks for a support or enterprise contract of an Open Source package, codify that the software will actually stay Open Source.

Whether you are a producer or a consumer of Open Source Software: Be mindful not to overextract value. If you do, the market may overcorrect.

“But we are starting to see a very disturbing trend where customers that everyone would recognize - they’re in the Fortune 50 - that are using NATS to power production-level services or functions or products, not only had never reached out for any type of commercial agreement with us, but actually have policies that say ‘If you’re in the CNCF and you’re incubating and graduating, we’re not paying for it, period.’” ↩︎

Go Big or go On-Prem

Mon, 02 Jun 2025 15:38:00 +0200

Probably my favourite thing about the “Cloud Exit” from Basecamp that DHH is so vociferously writing about is that it has shifted the predisposition about Cloud usage. For many years it seemed there was no choice: you created an AWS, Google Cloud or Azure¹ account and started creating resources. Who the hell is Dell? What’s a hypervisor?

But economic realities have kicked in and people are starting to wonder why they spend as much on their Cloud bill as it would cost to buy a full rack of servers while employing even more operations staff.

On-Cloud, Prem-Cloud

On-Prem never really went away; depending on the survey, it seems that at least half of computing workloads are still hosted in a private data centre instead of the public cloud. And of course, you might argue what “Cloud” even means: is it about the technology, the economic aspect, or culture? If you can autonomously provision your workloads on a cluster that’s managed by somebody else and is running on machines inside your organisation, it feels like Cloud to you, but has a pay-to-own cost structure.

Conversely, if your organisation uses the services of a Cloud provider, but you don’t do DevOps and software cannot be deployed without operator intervention, you might be spending Cloud-money without embracing the culture and reaping the benefits.

Oxide advertises themselves as “The cloud you own” as their product is a rack-scale computer that can be provisioned like a public cloud account, using e.g. Terraform. You don’t rent it, you buy it once and run it as long as it’s feasible – and I bet that’s longer than most people would expect.

And then there are combinations which never really made sense to me²: buying a bunch of servers and then paying heaps of yearly licence fees for every layer in the software stack – hypervisor, operating system, storage system, etc. Especially since none of those vendors is able or willing to solve your problems in case the shit hits the fan.

My biggest gripe is the low-levelness of many solutions that e.g. AWS offers: it takes astonishing complexity to get anything going. Configuring a setup on EC2 might be quite easy, but it means that you don’t get many of the benefits. You want to add a load balancer? Prepare to configure VPCs and subnets and manage multiple availability zones, and oh: don’t forget to add CloudTrail and CloudWatch!

Of course, with all these tools stacked up, your bill will too. Your trivial workload incurs a non-trivial cost. Also, most of the cloud setups I encounter are massively overprovisioned, just in case. It’s great that your development team doesn’t need to count CPU cycles, and can just write Python, but when you only utilise 10% of already expensive compute, you’re not getting any of the savings.³ Development environments could be shut off overnight, when nobody is using them, but getting this process in place seems to cost more than it saves in usage.

Even solutions like Elastic Container Service still require a lot of prefabrication: you don’t have to manage EC2 instances anymore, but now one has to figure out how to connect your container registry. It might make it easier to scale or to add redundancy, but the complexity is just moved around at best.

Kubernetes might be more provider-agnostic⁴, but I don’t think that anybody has ever made the argument that it is non-complex. And boy, do I see many setups that do not warrant its complexity. If the number of microservices outnumbers the number of developers, you already have a problem.

Above the Cloud

Some people said “fuck all of that” and started moving into more upstack, but lightweight solutions, like AWS Lambda. If your databases, storage and load balancers are already run by your cloud provider, you can go a step further and eliminate your internal complexity. Write a piece of JavaScript, Go or Python, upload it to AWS, and they host it for you.

As with everything, there are trade-offs: at the low end, cost is lower than spinning up any EC2 instance, but it can eat through any IT budget at the high end. You can start quicker, but you are locked into your provider. You can use a familiar programming language, but not necessarily any framework.

Heroku was truly ahead of their time: they certainly made mistakes, in technology as in business, but connecting your Git repo and getting your software deployed in minutes was truly groundbreaking. Unfortunately, in 2015 one half was building the next Instagram so they thought they’d outgrow Heroku within seconds and reached for more grown-up cloud solutions, the others were still SFTP-ing PHP scripts to their server. Maybe the world simply wasn’t ready yet.

It’s all about the Benjamins

The new generation of more integrated Cloud offerings has better chances: borrowing money isn’t as cheap as it was, companies don’t want to hire additional Cloud operators, and developers shall be more productive. People are reaching for more managed tooling: Vercel is already a big hit, despite the underlying framework, Next.js, being an absolute nightmare. Quality of integration is more important than quality of your individual components.

With services like fly.io, the operational burden stays low while offering a lot of options. If you’ve got a Dockerfile, you can get your service running in mere minutes and it’s almost trivial to scale to multiple regions while not being more expensive than AWS et al.

I absolutely get it when people do the maths and conclude that buying and running their own hardware makes sense. I also agree that paying for an integrated service is great. But shelling out top dollar for furniture that you need an expert to assemble and to keep together?

As I wrote in my previous piece, the big cloud providers have one enormous advantage: unified and centralised billing. CFOs across the globe still think they made a killer deal when the Cloud vendor offers them 20% off or gives them a five or six-figure discount, but these are not acts of charity – they just make a slight dent in the long run. Make no mistake: they still have obscene margins, and there is a lot of room for negotiation. Especially egress is a money printer.

If they don’t want to make you a seemingly generous offer, they either think that you’re not going to grow or that your organisation is too incapable to switch to a different vendor.

It’s much easier to add another ELB than to get the company credit card to sign up with a new provider. As a side note, if you are not letting your staff create new resources autonomously without an architecture review board getting involved, it doesn’t make sense to be in the Cloud at all. A strategy to prevent getting too complacent can be personal budgets for services that developers can use freely to spend with other providers without going through a massive purchasing process.

Computer hardware has become much faster and cheaper, but many of the providers do not pass the savings on to the customer. AWS used to reduce prices on a regular schedule, but those times are over. EC2 prices have at best stayed flat, while Amazon runs their hardware for much longer these days. It’s a marker of their success that they are dominating the market, but also that CTOs have become a bit blind to the fundamentals. Even I was stumped when I realised that the smallest VM instance at Hetzner now offers 4GB of RAM, at just under 4 EUR (USD ~4.50) per month!

It’s a matter of incentives, too: technologists are not really pushed towards being cost-efficient. Setting up a Kubernetes cluster may be driven more by the desire to be able to put it on the CV than to improve reliability or speed.

The technology sector, as unconventional as it can be, has largely kept sales quite traditional: lower base salary, bigger commission. Which kind of makes sense – when sales gets a win, the whole company profits. With technical roles, one could argue the same: when a developer cuts the Cloud bill, everybody saves. But all these savings will end up, at best, in the shareholders’ pockets. Only when it’s almost too late, management will start rampaging and trying to cut down, starting to put every line item in question.

I think a more balanced leadership approach needs to be adopted that emphasises more mid-term thinking: the long term is overvalued in my opinion – some C-level staff try to imagine a wild future vision, where they are AI-first or whatever makes them look smart or gets them through their next funding round. The short term is dominated by reactive firefighting. In between, you can push for a better discount with your Cloud provider, move some workloads either on-prem or onto servers you own, push for better utilisation or move to higher value, cheaper maintenance all-in-one providers. Employ people who are smart and adaptable, not those who just have the right things on the CV you are currently using. Chances are, your business will need to change more than it needs the current crop of tech.

AWS likes to emphasise in their training materials the dangers of “undifferentiated heavy lifting” in which you are doing things that don’t improve your product, but require your attention. If you look at the heavy lifting that you need to do to satisfy AWS’s well-architected framework, one could argue that you shouldn’t be a customer there.

How is your strategy going? Are you going more upstack or doubling down on on-prem? I’d love to hear from your experiences!

God help us ↩︎
Despite some people’s best efforts to explain it to me like I am five. ↩︎
A 10% utilisation also means that you are not paying a 5x premium on the compute – you are paying 50x! ↩︎
It isn’t though for most. ↩︎

Deutsche Cloud

Thu, 15 May 2025 16:23:00 +0200

Due to the unique political situation we are living in, there is an increased demand for European technology solutions. Call it derisking, sovereignty, or nationalism. Everybody seems to be at least pretending to look for new solutions from within.

The German IT industry is classically laggard: while the rest of the world is beginning to understand that the benefits of cloud computing are technological, not economic, in Germany, all the rage seems to be in public cloud these days. In my local government, everybody seems to have learned the word “hyperscaler” recently and doesn’t hesitate to repeat it relentlessly with limited understanding.¹

When Microsoft builds a data center in Germany, everybody seems to ignore all the nasty details and declares it EU-cloud, congratulating themselves on getting the deal.

A particularly strange public sentiment I’ve encountered is the admiration of the Schwarz Group IT division “Schwarz Digits” and their brand “STACKIT”. The Schwarz Group is the holding company of one of the biggest retailers in Germany. In 2022, they entered the limelight by starting a public cloud offering that they built based on their own needs and opened to the general public. A retailer building their own IT services and selling them as a service? How 2002!

I don’t mind people taking affairs into their own hands and building their own thing: buy hardware to run your software on? Sounds great, I’m in! But if you look at it for more than a few seconds, the details are massively underwhelming.

The service palette itself is a little bit sparse: aside from quite low-level features like virtual machines and block storage, they claim to run a handful of databases and other managed services. They offer only one message queue and it’s RabbitMQ, an odd choice.

There is no transactional email service, and their workspace offering is just Google Workspace in a trench coat.

The underpinnings are based on OpenStack: a sign of either cluelessness or insanity. I just hope they aren’t also paying for VMware.

Pricing is a bit opaque, despite their affirmation to be transparent, but the calculator produces prices somewhere between two and five times more expensive than the competition. All cloud providers are able to cut special deals with significant discounts, so real costs may vary. It’s certainly a different strategy than their retail one, where affordability is paramount.

The operational side doesn’t impress either. They currently have one region: “Heilbronn/Deutschland” or “eu01”, as their documentation likes to call it. They claim it consists of at least three availability zones (AZs), but then backtrack and say that the one region they operate has exactly three AZs.

Screenshot from STACKIT docs

Network and power look disappointing too: while Meta, AWS, and Google started building their own network gear ages ago, STACKIT doesn’t even seem to fully support IPv6. Their colocation offer hints at a very old-school 19-inch rack setup with a pitiful 2 kW supply for 47U. At least you get a three-year contract, that’s a good thing, right?

I am cherry-picking here, obviously, but the colocation market in Germany is fiercely competitive.

Still interested? You can’t just sign up and get going: you have to apply for an account in true bureaucratic fashion. If you’re outside of the German-speaking part of the world, you will be subject to extended interrogation. Even if you sign up with a valid VAT ID, your account creation will have to be manually approved first. Mine is still pending. (Update: My account application got rejected after they asked me to submit a credit report from my bank. I am worthy enough to have credit cards or finance a house, but not to open a public cloud account.)

I’m sure the next generation of developers will patiently wait to incorporate a company and wait for your confirmation email, instead of using Google Cloud’s free tier.

Screenshot from LinkedIn

In late 2024, they hired the former managing director of Google Cloud Germany to become the CEO of STACKIT. In a LinkedIn post, he described STACKIT as a “sovereign European hyperscaler”. Their “About Us” page, on the other hand, adds “Swabian”² and “German” to their labels. As if wrapping yourself in one flag wouldn’t suffice, they wrap themselves in multiple.

I wish them all the best, honestly. They run open source infrastructure to some extent and have opened up what could otherwise just have been an internal project. But the total lack of depth and the marketing focus on some self-defined sovereignty seems like a total miss. A narrative often presented in Germany is that Americans can advertise themselves very well without substance. I must say that some people in Germany could accidentally become number one in that regard.

It’s not 2002 anymore. There is plenty of competition on the market, some of it EU-based (EU regions of any public cloud), truly European (e.g., OVH), or German (e.g., Hetzner). There are more batteries-included solutions too (e.g., Fly.io). And increasingly, on-prem: with ever-larger cloud bills and the realization that owning instead of renting can be significantly cheaper, it will not be an easy battle for those who lack ambition.

I think a major strategic advantage of the big public cloud providers is (billing) integration: not even a vigilant CFO will mind a 20 Euro line item on a cloud bill, while an invoice from a better or cheaper new vendor may raise questions. Joining the cloud market requires a gigantic amount of capital: you need the bones, like VMs and storage, but you also need enough higher-stack services to satisfy the needs of a large number of customers. I think that Fly.io took the smarter route by partnering with select providers of high-value services, like Tigris or Upstash, while centralizing billing. Fly.io also own their hardware, but not their data centers.³

If you don’t compete on variety, depth, or value, what’s left is to be proud of your heritage. If someone on my street started offering lukewarm, stale beer for 10 Euro, I wouldn’t be proud, I would experience Fremdscham.

https://www.rhein-erft-kreis.de/aktuelles/meldungen/2024-Hyperscaler-Ansiedlung-in-Bedburg-und-Bergheim.php ↩︎
Swabia is a region in southern Germany, probably most famous for an unintelligible accent and penny-pinching abilities. ↩︎
They can still build data centers later, if they get wildly successful, since they own so much of their own stack. ↩︎

Keeping your OpenStreetMap Database Replicated

Wed, 16 Apr 2025 14:14:00 +0200

The physical world is changing all the time, and so is OpenStreetMap (OSM). Luckily, you can just tap the source by using the replication system. osm2pgsql can import your OSM data into PostgreSQL and also update from that replication stream.¹

If you want to scale out your database because you want capacity and/or redundancy, you need a strategy to orchestrate this process. One option is to run the import and update process on each Postgres node.

This has multiple drawbacks: due to the unique way the OpenStreetMap data format is structured, you need to keep a large state file to be able to update your imported database on each node.² Also, the update process itself is quite resource-intensive.

One importer, multiple replicas

A better alternative is to run one osm2pgsql instance that performs the initial import and continuous updates while all other PostgreSQL instances are read replicas. This can be performed using the built-in logical replication, which started shipping with PostgreSQL 10.

This offloads the update process to one node, while all replicas get a preprocessed Postgres logical replication stream, just applying the data updates.

To get your database ready for replication, a few steps are required. In the following guide, we assume that you want to publish the tables containing points, lines, roads, and polygons, which are required for e.g. rendering using Mapnik.

1. Replica Identities

Logical replication propagates changes to the downstream databases at a high level. To be able to apply an UPDATE, it must be clear which rows this applies to. Since logical replication, as opposed to streaming replication, can be partial – not all tables or columns or even rows have to be replicated³ – this requires a row identity.

Usually, you can just use a PRIMARY KEY, which most tables have. Alas, this is not the case with a default osm2pgsql import. You can either add one using a flex config or do it yourself:

ALTER TABLE planet_osm_point ADD COLUMN id SERIAL PRIMARY KEY;
ALTER TABLE planet_osm_line ADD COLUMN id SERIAL PRIMARY KEY;
ALTER TABLE planet_osm_roads ADD COLUMN id SERIAL PRIMARY KEY;
ALTER TABLE planet_osm_polygon ADD COLUMN id SERIAL PRIMARY KEY;

Then you can indicate to PostgreSQL that you want to use these as replica identities:

ALTER TABLE planet_osm_point REPLICA IDENTITY DEFAULT;
ALTER TABLE planet_osm_line REPLICA IDENTITY DEFAULT;
ALTER TABLE planet_osm_roads REPLICA IDENTITY DEFAULT;
ALTER TABLE planet_osm_polygon REPLICA IDENTITY DEFAULT;

2. Publication

You can now mark these tables as available for replication. Make sure that your wal_level is sufficient: it must be set to logical in your postgresql.conf.⁴

To create the publication:

CREATE PUBLICATION osm_pub
FOR TABLE
 planet_osm_point,
 planet_osm_line,
 planet_osm_roads,
 planet_osm_polygon;

3. Prepare for Subscribing

The logical replication facility only copies data, not the schema. Therefore, you need to dump the schema for the tables you want to replicate and create it on your downstream nodes.

Get the schema from your upstream:

pg_dump \
 -t planet_osm_point \
 -t planet_osm_line \
 -t planet_osm_roads \
 -t planet_osm_polygon \
 --schema-only > ddl.sql

Beware of TRIGGER and GRANT: these apply to your upstream but possibly not your read replicas.⁵ I use a little sed to remove them from the script:

sed -i '/^CREATE TRIGGER/ d' ddl.sql
sed -i '/^GRANT SELECT/ d' ddl.sql

Lastly, import the schema on your downstream(s):

psql -f ddl.sql

Your downstreams are ready to subscribe. You must be able to reach your upstream from your downstreams.

CREATE SUBSCRIPTION osm_sub
CONNECTION
 'host=your.upstream.local port=... user=... password=... dbname=...'
PUBLICATION osm_pub;

PostgreSQL will start importing table by table into your replica and then keep up with updates. The initial import may take a while.

You can observe the process from your PostgreSQL log:

logical replication apply worker for subscription "osm_sub" has started

And the pg_subscription table:

SELECT * FROM pg_subscription;
oid | subdbid | subskiplsn | subname | subowner | subenabled | subbinary | substream | subtwophasestate | subdisableonerr | subpasswordrequired | subrunasowner | subfailover | subconninfo | subslotname | subsynccommit | subpublications | suborigin
-----+---------+------------+---------+----------+------------+-----------+-----------+------------------+-----------------+---------------------+---------------+-------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-----------------+-----------
5555 | 1234 | 0/0 | osm_sub | 10 | t | f | f | d | f | t | f | f | host=10.1.1.2 port=5432 user=y password=x dbname=osm | osm_sub_1234 | off | {osm_pub} | any
(1 row)

If the upstream is unavailable, replication will pause and resume as soon as it becomes available again.

Closing Thoughts

I originally started writing this article five years ago when I first built this system, but I didn’t feel qualified enough to finalize it. The partners I built this with were much more experienced in streaming replication, but I stubbornly pushed for logical replication since the underlying mechanism has the aforementioned benefits and feels much more modern.

This approach definitely has drawbacks, too—most notably that you cannot promote a new upstream/primary, which is possible if you use streaming replication and something like repmgr. But since the database can be recreated at any point by just re-importing from source (which has the nice side-effect of getting rid of index bloat), this is quite acceptable in my opinion.

After shipping this to a customer, and also running it myself for multiple years, it has been running without adding any drama. I am only moonlighting as a DBA sometimes, so simple and effective solutions that don’t wake me up at 3 am are my favorites—and this is definitely one of those.

It must be noted that PostgreSQL is kind of a bad match for such workloads, and an OLAP-optimized DB would probably be a better fit, but PostGIS is extremely powerful when you want to work with geospatial data. I worked on geospatial systems that would make a database unnecessary, but losing access to PostGIS made that unfeasible – at least for my resources and skill level. ↩︎
Almost 100GB at the time of writing. ↩︎
If you already know exactly which tables or expressions you need on your replicas, you can restrict the data that will be published: adjust the CREATE PUBLICATION in section 2. ↩︎
https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-SETTINGS ↩︎
Triggers are used by osm2pgsql, but you do not need them on your read replicas, since these will not run the import/update process. Grants are for permissions, but your permission set on replicas may be different from the upstream. ↩︎

Typst is Weird and I Really Like It

Tue, 01 Apr 2025 15:55:00 +0200

For reasons not entirely understood to humankind, some software developers have an aversion to WYSIWYG: Typing into Microsoft Word¹ and immediately seeing what the document looks like seems strangely alien. In some cases, this is warranted: If you write a 300-page thesis, modern desktop software is not able to cope with it, while being perfectly able to play back 4K video that is being streamed live from some remote data centre.

As every self-respecting nerd in their adolescence, I experimented with LaTeX, a typesetting system that generates documents from plain-text input: it’s this promised land of perfect typography which has an aura of a scientific mindset. But to be honest, the whole thing is something for frizzy, grey-haired hippies: The syntax is horrifying, the tooling is byzantine, and adding a character that is not in the English alphabet can lead to a total meltdown.² Also, the typographical output is not as good as many people claim it to be.

For publishing on the web, Markdown came into dominance. And it’s not bad if your name is John Gruber and you never need to author a table. I do not want to be too harsh on Markdown; this post has been composed in it!

But it gets tricky when you write a “document”: be it a manual, tech documentation, an invoice or similar. You want a table of contents? Variables that you can reference? Insert a formula? Tough luck, you will be scripting.

Asciidoc has an acceptable syntax and promises more features, but it’s often unclear what is part of the language and what is part of one of the implementations. Tables of contents, for example, are handled very differently depending on whether you use Asciidoctor for generating an HTML file or Asciidoctor-PDF for creating a PDF. Especially the latter often feels half-baked.

I need to generate PDF documents from structured data on a regular basis, so I cobbled together a workflow consisting of YAML files for data input, a Jinja template that produces HTML, and then a little bit of Python code that combines those two and feeds it to WeasyPrint, which generates a PDF file. It works, but I don’t really love it.

A while ago, I stumbled upon Typst, which looked to me like an online editor for scientific writing but is, in fact, a typesetting system. It has a reasonable markup language, generates PDFs, is fast, and doesn’t litter your file system with intermediate files.

It allows a lot of scripting inside the markup, which makes it a perfect match for data-driven documents.

name: Jon Doe
sandwiches:
 - Chicken Teriyaki
 - Philly Cheese Steak
 - Salami

and

#let data = yaml(sys.inputs.path)
= Sandwich Documentation
Hi #data.name! Our records indicate that your favourite
sandwiches are
#data.sandwiches.join(", ", last: " and ").

glued together with a

typst w doc.typ --input "path=sandwich.yaml"

gives you a PDF document that looks something like this:

And while normal text looks very reasonable if you are used to the usual markup languages, you can use all sorts of tricks to generate a specific output. You have a number that you want to display as currency? Try this:

#let as_currency(value, round: true) = {
let v = value
if round {
v = calc.round(decimal(value), digits: 2)
}
let s = data.currency + "\u{00A0}" + str(v)
if s.ends-with(regex("\.\d")) {
s += "0"
} else if not s.ends-with(regex("\.\d*")) {
s += ".00"
}
s
}

If you spend too much time in Typst, your document may look more like a program than text sprinkled with markup, but that’s maybe what you wanted and what you’re most familiar with. A kind of strangeness that’s beautiful – and fun.

Better document processors are, of course, also available. ↩︎
WTF is {"o}? ↩︎

Journaling for Productivity

Thu, 27 Mar 2025 11:43:11 +0200

A technique that helped me focus is journaling, by hand, on paper. I can assure you that the first time I heard about journaling in a professional context, I must’ve rolled my eyes violently. But I’ve been doing it consistently for over a year, and I feel that it has finally stuck. I miss it when I leave my notebook somewhere.

The strategy is quite simple: I note the date and the project I am working on and start writing down what I’m doing or what I’m thinking about. When I feel that I am running in circles, I write down my conundrum. When I’ve achieved something, I write it down.

When I used to have a meeting, I’d pull out a sheet of paper, write down my notes, and then put it to the side. The effect was a stack of loose sheets with random notes. What meeting? When? Nobody really knows. My poor in-meeting notetaking skills, compounded with a total lack of structure, made it difficult to keep track. With a chronological log, it’s much simpler: There is a date on the sheet. I don’t lose it. When I flip through the pages, I know the mental context I’ve been in.

What I write down isn’t official or polished or ready to be shared with anybody, but this provides me with a certain kind of freedom. It allows me to forget, because it’s already written down.

Deciding on Something

“Something with A didn’t feel right, B wasn’t our preferred choice either, so I went with C,” is a common template for summarizing a decision. The more time that has passed since the decision, the murkier the arguments become.

Now, when I do solution discovery, I just write down everything in real-time. When I need to compile a report later or revisit my decision because some externality has changed, I go back to my notes.

It also takes away the pressure to remember non-critical details. As soon as it’s written down, my mind is allowed to drop it, and I can move on to the next thing immediately.¹ If a customer asks what I’ve billed for, I don’t have to reconstruct it by combing through emails or my Git commits; I can simply go through my journal log.

It’s not perfect; the sorting is chronological, and then there is a project title, but there is no search function – it’s just paper, after all. But I don’t really want something digital. I own an iPad Pro with a Pencil, but it’s another attention-grabbing device. I have looked at the reMarkable tablets, but my handwriting has gotten rather messy, and I don’t see the point in digitizing my notes unless it gets to a machine-readable state.

My paper notebook doesn’t need software updates, it can’t display notifications, I don’t have to pay a monthly fee, and the worst thing that can happen is that I run out of ink.

It took me a while to make it stick. It wasn’t a success right away, and it certainly takes discipline, like any other habit. For me, the right format and a nice pen made it more pleasant.² Compared to more radical productivity ideas like committing yourself to a monastery or waking up at five and taking cold showers, I can assure you it’s cheaper and definitely worth a try, but do give it a bit of time to sink in!

There is a concept called attention residue (Leroy 2009) that describes how switching to a new task is hindered by the previous one. Writing a thought down helps me to disengage from the old task. ↩︎
If you want to replicate my “setup”: It’s a LEUCHTTURM1917 Hardcover Medium (A5), Fox Red, checkered and a Parker Jotter Stainless Steel with Golden Finish. ↩︎

Getting Back into Writing on My Own Terms

Thu, 13 Mar 2025 13:11:00 +0200

For the last five years, I’ve probably been writing more than ever, which you wouldn’t guess based on my publication schedule on my very own website (spoiler: it’s been four years!). Most of it has happened internally somewhere: multi-page proposals, architecture concepts, or just pull request texts and Slack messages of epic length.

In the last few years of my work, there were only a couple of weeks when I spent working with somebody at the same place at the same time. Some collaboration happens on calls, and these can be really productive for actively collaborating on some code or nailing a particularly nasty bug. Nevertheless, I feel that for aligning, advocating, or documenting, they tend to have a high signal loss. Recording meetings can feel like a solution, but someone will not revisit an hour-long meeting by watching a video; they will rather hit you up on Slack, ugh.

Text, on the other hand, is easily searchable, can be stored along with code, rewritten as needed, and stored for less than a penny.

Why?

One of my favorite methods to drive documentation is asking why relentlessly: Why do we encounter this issue? Why are we solving it? Why is this the approach we took?

Instead of writing a commit message or pull request that explains what has been done, I write why those changes were necessary. The what can probably be summarized by some babbling LLM or, more optimally, by looking at the content. The same applies to code comments: code shouldn’t really need comments about what is being done, as this should be self-explanatory by virtue of naming and structure. A start.before(end) makes an inline comment unnecessary, which would be needed for code containing copious amounts of boolean algebra.

The aspects of why in code are also much less prone to getting stale than the what. Often, implementations change over time, but the comments stay, leading subsequent readers astray.

The why also makes it easier to spot your own bullshit. If you propose an architecture and the why starts to feel awkward and long-winded, you’re probably offering the wrong thing, and it gives you the opportunity to catch it before somebody else does.

Diagram [noun], ugly image

I think this explains my skepticism about architecture diagrams: they mask everything besides the what. They reduce complex plans into a 2D plane while losing all the fidelity that makes such a diagram necessary in the first place. Even simple assumptions about time relationships in diagrams can break down, which happened to me when I got a diagram created by somebody native in a right-to-left language. After a call, it turned out that I did not need to build a machine that reverses time; I just started to read from the wrong side.

Often, architecture diagrams create an aura of “the difficult grown-up work has been done; now draw the rest of the fucking owl, you coding monkey.” I’ve been guilty of such diagrams myself, one so horrifying that I cannot bring myself to offer it to the general public, as it may make me liable for people being sick. It helped me think about a problem in time-space, but it didn’t offer any sufficient explanation to anybody else. Just recently, I revisited it to re-familiarize myself with it after several years, but I had to read the code since this was much easier to parse than my own drawing.

 ┌─────────────────┐
│ Transport (net) │
├─────────────────┴──────────────────────────────────┐
│ ┌───────────────┐┌───────────────┐┌───────────────┐│ ┌──────────────┐
┌──│ │ REST ││ Websocket ││ STUN/TURN ││──┐ │ Background │
│ │ └───────────────┘└───────────────┘└───────────────┘│ │ └──────────────┘
│ └────────────────────────────────────────────────────┘ │ │
│ │ │
│ ┌──────────────────┐ ┌──────────────────┐ │ │
└─────▶│ Models │◀─────│ Controller │◀─────┴────────────┘
└──────────────────┘ └──────────────────┘
│
┌────────────┴─────────────┐
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Database │ │ External │
└──────────────────┘ └──────────────────┘

I do not have a fundamental issue with diagrams, I think. Sometimes I even sprinkle some diagrams into code¹, but it has to be like a table of contents, not a pretend-authoritative piece of imagery that inevitably drifts from reality.

If you ever tried to generate diagrams from code, you will have realized that they aren’t as pretty or as useful as you hoped. I don’t think it’s a matter of technique, but rather that the reality of systems is more messy than architecture design wants it to be.

Even AWS fell into that trap with CloudFormation Designer, which was supposed to be a simple way of architecting your cloud systems by drawing diagrams. It feels natural at first sight: you drag and drop high-level, prefabricated components onto a canvas. And while the declarative aspect definitely took off with the likes of Terraform, diagram-based design got mostly abandoned.

Text is just so unreasonably effective. And that’s despite the fact that mass literacy is a relatively recent phenomenon: it took until the 1970s for the majority of the world’s population to be considered literate².

Even if the end product isn’t the text, but a machine, a software, or a YouTube video explaining the merits of a reasonably priced semi-offroad motorcycle, you just know when the writing is good.

Monodraw is an amazing piece of software for creating diagrams suitable for among-code documentation. ↩︎
https://ourworldindata.org/literacy ↩︎

Don't UUID Yourself

Fri, 10 Dec 2021 12:37:00 +0000

For a long time, sequential IDs were good enough for everything. Just use AUTO_INCREMENT or SERIAL, job done, right? But then you realize that your competitor might now figure out how many customers you have, because they just have to create a new user and look at their ID. Or someone might enumerate all of your customers, starting at one and adding one until they are finished.

As an alternative UUID (esp. UUID v4) has become increasingly popular and the reasons are pretty obvious: It’s a 128 bit space, so guessing an ID is much harder, they are not sequential, so you can’t enumerate them. And most importantly, it makes your software look so much more enterprisey.

UUIDs incorporate a trade-off that they are made for machines, but are also human-readable. But I’m not sure that they are always the right choice.

128 bit are just 16 byte, but due to the hex encoding and the separation into groups, they’re in fact 36 characters long. Thus they have a 125% overhead, but that’s not too bad when you consider that they are suitable for human consumption. But about the last point: I find it rather difficult to skim through lists of UUIDs:

They don’t give any indication about the kind of data. When you’re developing or debugging you try to make sense and find patterns, but UUIDs to - at least - my brain are mostly noise.

Also it may be difficult if you have a large collection and start to get UUIDs that look similar. The following IDs might look equal:

c0d72339-f3fd-438f-934f-283821683226
c0d72339-f3f9-4e8f-934f-283821683226

but they’re different. In a heated debugging situation this might turn out to be very annoying.

Could there be an alternative?

If you want to keep human-readability and have a similar address space, an alternative might be random, prefixed alpha-numeric strings. Prefixes help distinguish different kind of identifiers: user IDs could be prefixed with “usr-”, payment IDs with “pay-” and so forth.

A 6 bit per ASCII char alphabet consisting of ABCDEFGHJKLMNPQRSTUVWXYZ23456789 is case insensitive and even eliminates confusing characters like 0 and O as well as 1 and l. To match at least 128 bit of UUID, you’d need 22 chars in that alphabet. Adding a type-specific prefix, with a length of four chars (as in the example above), would result in 26 chars instead of the UUID’s 36. Such a string would be more readable and distinguishable and also have a smaller overhead.

If you’re already using UUIDs, stick with them, they’re nice. But when you start off with something new consider if you want to add nine SLOC to your code:

import "math/rand"

const alphanum = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789"

func randomAlphaNumToken(prefix string, length int) string {
 b := make([]byte, length)
 for i := range b {
 b[i] = alphanum[rand.Intn(len(alphanum))]
 }
 return prefix + string(b)
}

:= Need Not Allocate

Thu, 10 Jun 2021 13:35:00 +0000

From time to time I work with planet-wide geospatial data, which in itself is nothing new, but I was in discontent with the ingestion stack I was using, so I built a stream processor. Everything was going well, my prototype was around five times faster than the software I was previously using. Then, I realized the classic GIS mistake: I forgot to reproject the coordinates.

For those unfamiliar with projections: It’s just a way of describing coordinates on a globe. Projections define numeric boundaries and can describe a geographical region in different ways, e.g. when you need equal distances or equal angles in a region. It depends on the context which projection you’ll want to use.

My source data came in EPSG:4326, but I needed EPSG:3857 for rendering. Transforming the coordinates is a trigonometric function with some constants, but doing this for billions of values takes some consideration. The mathematical transformation is quick, but allocating is not.

Go doesn’t make you think too much about allocation: Slices grow automatically, memory is acquired as you go and garbage is collected. I’ve seen some very high traffic Go applications that run smootly with a small memory footprint without anyone having spent any thought on memory optimization. But every luxury has its limits: When you deal with long data streams, allocations can stack up.

I knew it wouldn’t be viable to decode the coordinates from my stream format, put them into a slice and recode them again, so I had to do some in-place trickery, resulting in this code:

func (f *Feature) projectPoint() {
 offset := f.geometryPos
 floatX := math.Float64frombits(binary.LittleEndian.Uint64(f.Geometry[offset : offset+8]))
 floatY := math.Float64frombits(binary.LittleEndian.Uint64(f.Geometry[offset+8 : offset+16]))

 floatX3857 := degToRad(floatX) * earthRadius
 floatY3857 := math.Log(math.Tan(degToRad(floatY)/2+math.Pi/4)) * float64(earthRadius)

 binary.LittleEndian.PutUint64(f.Geometry[offset:offset+8], math.Float64bits(floatX3857))
 binary.LittleEndian.PutUint64(f.Geometry[offset+8:offset+16], math.Float64bits(floatY3857))
 f.geometryPos += wkbPtSize
}

If you’re sensitive to allocation, you might look at those “:=” and have an urge to elimiate those by inlining those operations. But beware, this makes the code less readable and in fact, doesn’t have any performance benefit.

A small benchmark confirms this: projectPoint() doesn’t allocate at all, because the compiler knows that nothing will escape.¹

In short: Think about the memory, don’t reallocate slices if you don’t need to, but check if the compiler is maybe already smart enough, so you don’t need to compromise readability.

Use b.ReportAllocs() in your benchmark runs. ↩︎

Load Balancing Without Giving Away the Keys

Mon, 31 Aug 2020 15:49:00 +0000

Distributing the load of a web application or especially an API endpoint using a load balancer (LB) is highly useful: You can get better performance, make software rollouts smoother and withstand node failure.

If an application node becomes unavailable due to failure or a planned maintenance, the LB notices that it does not respond and stops send traffic to it.

The LB itself can go down, too, but since LBs can be made stateless, failover and redundancy can be achieved more easily through e.g. the usage of CARP¹ or anycast. Many cloud providers offer managed load balancers.

Since HTTPS has become the standard even for non-private data transfers, the management of TLS certificates and keys needs to be considered as well. Often software operators opt to decrypt the TLS on load balancer level and send traffic via unencrypted HTTP to the nodes, compromising confidentiality.

TLS can also be handled by the nodes itself, but this bears two challenges: All the nodes need the same key and certificate and also, if you’re acquiring the certificates from Let’s Encrypt or any ACME-compatible CA, they need to serve the appropriate authorization challenge.²

For serving HTTPS directly from a Go application, there is the certmagic library, which also powers the automatic TLS management in Caddy. By default, it manages key generation, cert acquisition and wraps the HTTP server with TLS. The storage for the keys is rather flexible: Normally they’re stored on disk, but you can add a different adapter or write your own. There is a third-party S3 adapter, which allows you to store your certs/keys/challenges on Amazon S3. If you configure your nodes to use the same S3 bucket, it doesn’t matter which node will be asked by the load balancer to serve the challenge, all of them can answer, because they’re pulling the challenge via API.

Personally I found it unacceptable to store all the keys in plain text on some else’s hard drive. Perfect forward secrecy prevents from decoding HTTPS traffic before a breach, but with a third party service it’s impossible to determine whether such an event took place.

This is the reason why I built a custom certificate storage for certmagic, which stores the keys, certs and challenges on any S3-compatible store (e.g. Backblaze, Digital Ocean Spaces or your own Minio instance), but encrypts it using NaCl’s secret box³ before sending it off.

You don’t have to trust any of the provider’s claims about ”at rest“ encryption and you don’t have to implement any highly available storage yourself. You just have to spend a few pennies per month for any S3 storage, without lock-in.

This allows you to do HTTPS without having to break encryption mid-way and you can offload TLS to the nodes, instead of the LB having to do all the work.

FreeBSD Manual on CARP ↩︎
To be pedantic, only if you’re using either TLS-ALPN or HTTP authorization. You can also use DNS validation, if your DNS software or zone provider supports this. ↩︎
More specifically, the secretbox implementation in Go ↩︎

Printing with SwiftUI

Wed, 08 Jul 2020 16:12:00 +0000

This post combines one of the world’s oldest information technologies with one of the newer ones: Printing on paper and SwiftUI. SwiftUI is Apple’s rather new framework to declaratively design user interfaces, which runs on all their platforms. It has a few rough edges, but it is very powerful, because it allows mixing and matching with classic NSViews on macOS, so you can even use them for print (or PDF) output.

macOS has always had rather great printing support and for a first-class Mac app this is a feature one shouldn’t miss.

Integrating the print workflow into a new app is not too hard and starts with connecting the “Print…“ menu item’s action to an outlet. The menu configuration is not part of SwiftUI, but lives by default in a dedicated storyboard Main.storyboard.

With an option click drag (aka right click drag) you can connect the print menu entry to an outlet in your AppDelegate class.

Your delegate will look something like this:

@IBAction func applicationPrint(_ sender: NSMenuItem) {
 ...
}

Inside this method you’ll need to setup your view that you will want to print, set up the Print layout info and then hand over to the user to make a decision about the output format and device.

First of all, inside your delegate, set up an NSPrintInfo instance and configure it:

let printInfo = NSPrintInfo()
printInfo.scalingFactor = 0.7

I used a smaller scaling factor, because the font sizes suitable for screens were too large for printing in my particular case.

Unless you want to print the whole window with e.g. all input elements and text fields (in which case you can simply pass window!.contentView to the print routine), you need to initialize your printable view and populate it with some data. Let’s assume you have a SwiftUI view called EntryListView and it takes a list of string as data:

let entryListView = EntryListView(data: ["This is a string", "And another one", "One more"])

Since this is a standalone SwiftUI view, it needs to be contained in an NSHostingView:

let printContainerView = NSHostingView(rootView: entryListView)
printContainerView.frame.size = CGSize(width: 800, height: 600)

This view is now wrapped and can be passed to a print operation alongside the print info configuration:

let printOperation = NSPrintOperation(view: printContainerView, printInfo: printInfo)
printOperation.printInfo.isVerticallyCentered = false
printOperation.printInfo.isHorizontallyCentered = false
printOperation.runModal(for: window, delegate: self, didRun: nil, contextInfo: nil)

If the user selects “Print…” from the menu bar or hits “⌘P”, a print dialog will popup with your view ready to be printed:

Migrating Foreign Keys in PostgreSQL

Thu, 21 May 2020 08:30:48 +0000

Applications often need to work with external IDs, e.g. UUIDs of third party services or SKUs. In such cases, an external ID should be stored just once in a mapping table and from there on only referenced by an internal (e.g. serial) foreign key. Without a dedicated mapping table, you will carry such external identifiers through all tables. This may seem reasonable at the beginning, but may soon cause trouble, e.g. if you want to add a second type of external service or when you realize, that those seemingly unique identifiers are not as unique as assumed.

Even though I should know better, this kind of mistake happens to me from time to time anyway. Or it happens to others and I have to save the day.

Because PostgreSQL is not the wild west, if you reference a field in a different table, this will be enforced through a constraint, which is fantastic for data consistency, but annoying if you misdesigned your data structure and have to change your foreign keys. But, don’t worry, it’s still possible to change this inside a single transaction with those four-ish steps:

Create the new key ID column, make it unique (e.g. serial)
Identify all tables which reference the (old) ID:
1.1. Note the table name
1.2. Find the referencing column
1.3. Determine the foreign key constraint name (see pg_constraint table)
For each referencing table:
2.1. Drop the existing constraint (from 1.3.)
2.2. Update the referencing key values to the new ones
2.3. Add new constraint for the new base table ID
Drop primary key constraint from base table
Create new primary key constraint on base table

Example

Let’s assume your application tracks users, which have external identifiers. At the moment you got a user table, which has this external ID as the primary key:

`user`

external_user_id (primary key)	user_name
12312	Alice
91823	Bob

Thus, all tables that reference any user have those long, external identifiers:

`user_permissions`

user_id	can_read	can_write
12312	true	true
91823	false	true

Now, if you want to gain flexbility and add an internal ID, you can start off with adding a serial column to users:

ALTER TABLE user ADD column id serial UNIQUE;

user would now look like this:

external_user_id (primary key)	user_name	id
12312	Alice	1
91823	Bob	2

Now, for every referencing table, you need to strip the constraint, update the values to the new serial ID and then add the new constraint:

ALTER TABLE user_permissions
 DROP CONSTRAINT user_permissions_user_id_...id;

UPDATE user_permissions
 SET user_id = (
 SELECT id
 FROM user
 WHERE external_user_id = user_permissions.user_id
 );

ALTER TABLE user_permissions
 ADD CONSTRAINT user_permissions_user_id_...id
 FOREIGN KEY (user_id)
 REFERENCES user (id);

user_permissions looks now like this:

user_id	can_read	can_write
1	true	true
2	false	true

Finally, you can now change user.external_user_id without constraints, you just have to swap out the primary key in user:

ALTER TABLE user DROP CONSTRAINT user_pkey;
ALTER TABLE user ADD PRIMARY KEY (id);

At this moment, you can even put the external identifier into a different table, if you wish, thus possibly separating the concerns even better.

Bonus Content for Django Users

Swapping out primary keys is too difficult for the automatic Django migration assistant, so you’ll need to get your hands dirty. Obviously, you can do all those steps as raw SQL migrations, but then makemigrations will still nag you about changes in your model, thus you got to use SeparateDatabaseAndState, apply the changes with SQL and then tell Django what the effect is. For an example, see my code in osmcal.

Extra Bonus

If you’re working with raw SQL migrations, you might run into trouble and all you’ll get is some context-free SQL error message. For better debugging, you can use the sqlmigrate subcommand which will display the SQL queries instead of directly executing them. That way you can step through the statements and debug them step by step.

./manage.py sqlmigrate <app name> <migration name>

Making of the OpenStreetMap Calendar

Fri, 15 May 2020 12:28:17 +0000

Wikis have changed the landscape of information technology: No matter what organisation you‘re in and what topic you‘re working on, chances are that there is a Wiki, in which you can collaborate with others. Mostly without asking permission, you can put in notes, make lists and refine the material of others. Same with the OpenStreetMap wiki: Some pages are the standard source of information, like “How to map a…”, others are things relevant to just a single person, like pages where mappers list all the street they‘ve already mapped. And then there are things that have started with a good idea and gotten out of hand…

While collaborating on texts is good in most Wiki software, when it gets to tables and templates and structures, things get confusing, rather quickly. Sadly one of those examples is the Wiki calendar, which involves fiddling around with the custom templates, writing markup and someone rotating past events by hand. When we were talking about the OSM Wiki during events, people have been complaining repeatedly about the calendar. Some communities don‘t even announce their events in it, because they have shifted to Meetup or Facebook for better usability. There have been discussions how to solve it, maybe some sort of plugin or extension, but nothing has happened.

Inception

So, in early summer of 2019 I was mad as hell and couldn‘t take it anymore, so I sat down and started writing a standalone, special purpose web-based calendar for the OSM community. With maps!

Initially I wanted to utilize the infrastructure of the German OpenStreetMap Local Chapter, FOSSGIS, but they were busy generating rules about what I got to provide before getting a precious openstreetmap.de subdomain, so I went on my own and the domain osmcal.org was born.

The software is based on the incredibly powerful Django framework which helps with HTML templates, form handling, databases, authentication and much more. It‘s all open source, of course, as it should be. The issue tracker is on GitHub and already attracts a small community of people with bug reports, feature requests or general questions.

Progress

While osmcal started with a minimal feature set, development has continued and with the input from community members some features have shipped since then:

In October 2019, the possibility to “join“ events has been added, so organizers get a list of participants and as a participant you can see who else is going to come
In December 2019, iCalendar subscriptions have been made available
Since February 2020, event organizers can create surveys for participation, so they can ask people for e.g. their t-shirt sizes, their dietary preferences or skill level.

Special thanks to all reporters, especially @jbelien, @adrienandrem and @qeef for their collaboration and testing!

Future

Of course, at the moment, in-person meetings are out of the question, but this doesn’t mean that there are no events at all. Some are postponed, but many are happening online and those need to be scheduled and communicated as well. Thus, now it’s about time to integrate time zone support, which I initially deemed unnecessary as every event would have a location, but that has been disproven by the increased need of online events.

Also, I want more communities to integrate osmcal into their websites and wiki pages, so their lives get easier by not having to create and maintain their events on multiple platforms.

I was asked to consider applying for an OpenStreetMap Microgrant, so I submitted the aforementioned ideas with a request for funding. If you like to see those features and want to support further development, consider expressing your support on my grant application.

Making more connections

One new functionality that will be coming soon is community tagging, so organizers can create non-OSM-related events and have a dedicated feed with all their events. But more on that when it’s done. Stay tuned.

Faster Map Making With osmium

Sun, 22 Mar 2020 17:29:08 +0000

It’s sunday and I am fiddling around to make some maps, maybe to put them up my own wall and that’s the moment where I usually stuggle to get data out of OpenStreetMap into QGIS: Overpass is slow and won’t give you too much data, Shapefiles are ok, but this only works for limited regions and filtering and exporting OSMPBFs into GeoJSON is annoying.

Of course it would be better to have the data in a local PostgreSQL, but in order to get it there, you need to think about what data you want to load, but since you’re still iterating and fumbling with the data, this is not yet clear, so you’d need to have some cycle of loading, purging and reloading again. Annoying.

Then I remembered that you can use osmium to get all the data into PostgreSQL, without having to filter, which is not fast enough for tile servers, but easily fast enough for some experimentation.

Just get osmium tool and fire up:

osmium export -f pg -o taiwan.pg taiwan-latest.osm.pbf

Create a Postgres database and a table for the data:

CREATE DATABASE mapmaking;
\c mapmaking
CREATE EXTENSION postgis;

CREATE TABLE taiwan (
 geom GEOMETRY,
 tags JSONB
);

And now suck the data into the table:

\copy taiwan FROM 'taiwan.pg'

Now you got all the geometries in one column, the other holds all the tags. Now, you can fire up QGIS and use that table with some rudimentary querying action:

SELECT
 geom, tags->>'ele' AS ele,
 tags->>'name',
 tags->>'name:en' AS name_en
FROM
 taiwan
WHERE tags->>'natural'='peak';

Enjoy!

By the way, my work-in-progress greyscale map looks like this: