:= Need Not Allocate

From time to time I work with planet-wide geospatial data, which in itself is nothing new, but I was in discontent with the ingestion stack I was using, so I built a stream processor. Everything was going well, my prototype was around five times faster than the software I was previously using. Then, I realized the classic GIS mistake: I forgot to reproject the coordinates.

For those unfamiliar with projections: It’s just a way of describing coordinates on a globe. Projections define numeric boundaries and can describe a geographical region in different ways, e.g. when you need equal distances or equal angles in a region. It depends on the context which projection you’ll want to use.

My source data came in EPSG:4326, but I needed EPSG:3857 for rendering. Transforming the coordinates is a trigonometric function with some constants, but doing this for billions of values takes some consideration. The mathematical transformation is quick, but allocating is not.

Go doesn’t make you think too much about allocation: Slices grow automatically, memory is acquired as you go and garbage is collected. I’ve seen some very high traffic Go applications that run smootly with a small memory footprint without anyone having spent any thought on memory optimization. But every luxury has its limits: When you deal with long data streams, allocations can stack up.

I knew it wouldn’t be viable to decode the coordinates from my stream format, put them into a slice and recode them again, so I had to do some in-place trickery, resulting in this code:

func (f *Feature) projectPoint() {
 	offset := f.geometryPos
 	floatX := math.Float64frombits(binary.LittleEndian.Uint64(f.Geometry[offset : offset+8]))
 	floatY := math.Float64frombits(binary.LittleEndian.Uint64(f.Geometry[offset+8 : offset+16]))
 
 	floatX3857 := degToRad(floatX) * earthRadius
 	floatY3857 := math.Log(math.Tan(degToRad(floatY)/2+math.Pi/4)) * float64(earthRadius)
 
 	binary.LittleEndian.PutUint64(f.Geometry[offset:offset+8], math.Float64bits(floatX3857))
 	binary.LittleEndian.PutUint64(f.Geometry[offset+8:offset+16], math.Float64bits(floatY3857))
 	f.geometryPos += wkbPtSize
}

If you’re sensitive to allocation, you might look at those “:=” and have an urge to elimiate those by inlining those operations. But beware, this makes the code less readable and in fact, doesn’t have any performance benefit.

A small benchmark confirms this: projectPoint() doesn’t allocate at all, because the compiler knows that nothing will escape.¹

In short: Think about the memory, don’t reallocate slices if you don’t need to, but check if the compiler is maybe already smart enough, so you don’t need to compromise readability.