Unexpected GC Pauses in Go

We recently had an opportunity to compare the garbage collection (GC) behaviour of a small Go application at different heap sizes. This application was built to migrate data away from a legacy datastore. The application contains an in-memory cache which can be configured to clear itself at different rates. This means we can look at the command during a GC cycle at various heap sizes and compare the impact of this on GC behaviour.

We found unexpected and severe whole system pauses during the garbage collector’s mark phase. This was an surprising, as Go widely advertises itself as having a low pause garbage collector. In particular we found that the expected ‘stop the world’ pause was very short, but that whole system pauses experienced during the mark phase were several orders of magnitude longer. Observing that these pauses exist is important for understanding why we see such a large performance impact from GC cycles when there are plenty of CPU resources available and the reported pauses are very short.

In this post we will detail what we saw during this experiment and discuss how this impacts the way we investigate GC performance for all of our systems going forward.

It is worth noting that the system measured here was compiled with Go version 1.10. As the GC algorithm is under constant development these effects may disappear in later releases.

Roleman Part 1: Why We Created Roleman

Recently, I wrote a PL/PGSQL extension which provides some basic functions for creating and altering roles, and managing permissions. The extension was built to improve our tooling for PostgreSQL user creation internally. Since this has large number of external applications, it has been released to the public under the PostgreSQL license.

This blog post is the first in a series on this extension. In it I cover the difficulties which come with creating tooling around utility statements in PostgreSQL as a whole, why centralising this in user defined functions is a good idea, and what kinds of problems we are trying to solve.

This Programmer Tried to Mock an HTTP/2 Server in Go and Here’s What Happened

Testing has always been a must at Adjust. Serving tens of thousands of requests per second, we can’t afford tiny oopsies caused by some pointer being null when we didn’t expect it or a good old side effect in a presumably clean function. Tests are considered first-class citizens in our code base and are mandatory for any pull request that adds or changes functionality.

Just like tests, any dependency that we bring into our projects becomes a part of our code base. Since no one likes maintaining code written by some unknown dude (do you?), we try to reduce the number of third-party packages we use to an absolute minimum and prefer writing or own tailor-made libraries.

This is why we decided to stick with the minimalistic but decent testing package that Go provides as a part of its standard library.

HTTP Streaming in Elixir

One of the Elixir web apps we have at Adjust acts as an API gateway — it receives a request, applies authorization and/or authentication and then passes the request along to an actual requested service. As you probably know, if you’re building microservices, this approach is quite common. In our case, the API gateway also stands before a service that is responsible for generating reports. These reports are usually big json or csv blobs and sometimes they are as big as a few hundred megabytes. Because of this, downloading such a report via API gateway just to send it to a client does not sound like a good idea. Below, you can see what happened to our naïve implementation when a number of clients were trying to download sizeable reports.

screenshot

In this blogpost, I’d like to describe how we’ve implemented transparent streaming of HTTP requests directly to a client.

FastHTTP Client

At adjust we recently tried to replace the Go standard http library with fasthttp. Fasthttp is a low allocation, high performance HTTP library, in synthetic benchmarks the client shows a 10x performance improvement and in real systems the server has been reported to provide a 3x speedup. The service we wanted to improve makes a very large number of HTTP requests and so we were very interested in using the fasthttp client.

In the course of making the switch we encountered a number of difficulties. First the fasthttp library presents a very different interface to the programmer which must be adjusted to. Second there were a number of quirks in the implementation which made progress rather slow.

How We Deploy Elixir Apps

We at adjust recently started to use Elixir. We built a couple of small services using the Phoenix framework which successfully went live. In this blogpost I’d like to talk about, I’d say, the most undiscussed topic when it comes to Elixir — deployment.

Dive Into Deep Linking

In the mobile world, deep linking is a technology that launches an app and opens a specific page once a user clicks a URL on a web page or in another app. We will dive into the details of implementing deep linking for your app in this article.

Istore: PostgreSQL Documents for Analytical Workloads

Inspired by the PostgreSQL key/value data-type hstore, we developed the istore extension with support for operators like + and aggregates like SUM for semi-structured integer-based data.

While the hstore allows arbitrary textual-data as its keys and values, in an istore document both keys and values are represented and stored as integers. Therefore istore fits nicely in an analytical workload. User journeys, cohort or funnel data, distributional data and many other scenarios can be efficiently modeled and stored in PostgreSQL using istore.

The extension comes with two data types: istore and bigistore, the former having int and the latter bigint as values; keys being int for both. This article demonstrates the efficiency of istore and some of its applications through two examples - aggregating logs and analyzing event funnels.