At adjust we recently tried to replace the Go standard
http library with
fasthttp. Fasthttp is a low allocation, high performance HTTP library, in synthetic benchmarks the client shows a 10x performance improvement and in real systems the server has been reported to provide a 3x speedup. The service we wanted to improve makes a very large number of HTTP requests and so we were very interested in using the
In the course of making the switch we encountered a number of difficulties. First the
fasthttp library presents a very different interface to the programmer which must be adjusted to. Second there were a number of quirks in the implementation which made progress rather slow.
Making a simple request
To begin with we would like to learn how to perform a simple HTTP request using the
fasthttp client. Below is a very simple request using the Go standard library, error handling has been omitted for brevity.
For all the code snippets below the test server writes the request’s “User-Agent” header value and body into the response on separate lines. We write the actual output of the snippet in comments beneath each print statement.
1 2 3 4 5 6 7 8 9 10 11
fasthttp request can be written, also without error handling, very similarly:
1 2 3 4 5 6 7 8 9 10 11 12 13
Getting the response body
The body of an
http.Response is exposed as an exported
io.ReadCloser field. The body of a
fasthttp.Response is exposed via the
Body() method call which returns a
byte. The implication of this is that the entire body must be read and a sufficiently large
byte allocated before the body can be processed. This is a surprising feature of a library which prioritises performance and low memory allocations.
One curious aspect of the
Body() method is that it returns no error, in contrast to reading from an
io.ReadCloser. It would be interesting to see how that method is implemented to get a better idea of how
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
Body() method operates on two unexported fields
bodyStream. It first checks if
bodyStream is non-nil, and if it is, reads from the
bodyStream into the
body field. Finally the contents of the
body field are returned to the caller.
This is pleasantly straightforward, but there is one odd wrinkle, this method will silently eat errors.
Looking at line 15 in the example above we can see that any errors encountered while reading from
bodyStream are written into the
body field and the original error is not returned. An error could occur, but we would never find out about it. Lets look further into our simple HTTP request example to see how the
Body() method would actually execute.
If we trace the execution of our simple request above we find the following execution path:
1 2 3 4 5
ReadLimitBody(...) we find this critical piece of code
1 2 3 4 5 6 7
We can see, on line 3, that the call to
readBody(...) sets the bytes
bodyBuf.B to be the result of reading from the connection. So the stream reader field will be nil. We can see that errors are being returned from the
readBody(...) method call. That’s good, but we have only covered one simple case. From further analysis I do believe that errors are not swallowed by the
fasthttp client, but I am not certain. There is a potential execution path which results in errors being silently swallowed.
Making POST requests
Our existing application performs both GET and POST requests. We ran into a small problem making POST requests. We will start with a simple POST example using
fasthttp. Here we set our method to POST and fill in the body with some form-encoded values. Now we see both the “User-Agent” and the non-empty request body in the response.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Setting your “User-Agent”
Next we want to set our “User-Agent” header manually, but there is a small problem.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
While the standard library
http.Client does provide a default “User-Agent” header value, this value is overridden when any other value is provided. Fasthttp is still sending it’s default “fasthttp” and our “Test-Agent” value is not being picked up.
We wanted to get a better look at the headers that were being set, so we added a single debug line
println(req.Header.String()). Now we can no longer ignore errors in our code, because that innocent looking
client.Do(...) to fail.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
When we print the request headers we get to see the preloaded “User-Agent: fasthttp” header value is still stored, and in particular ahead of our “Test-Agent” value. This certainly explains why we aren’t seeing our value. We will look into why this is happening after we deal with the request error.
Why do our requests return an error?
After adding a simple
println statement we now get the error “Error: non-zero body for non-POST request. body=”p=q”“. The client now seems to believe that our request is not a POST. The critical call path here is
1 2 3 4 5
We can look into the
IsGet() method to see some interesting caching behaviour.
1 2 3 4 5 6 7 8
The method IsGet() reads the
RequestHeader.method field and sets the
RequestHeader.isGet cache field, to speed up future method calls. Unfortunately at this point we haven’t set our method and in the absence of any value it defaults to GET. So
RequestHeader.isGet is set to true, which causes future calls to
IsGet() to return true regardless of the value the
RequestHeader.method field. Critically this method is also called inside
HostClient.doNonNilReqResp(...) to test whether the request should have an empty body or not, causing the error we see above.
It’s worth noting that the call path contains 4 exported methods, any one of which would create the same confusing behaviour. You must be very careful to call
req.Header.SetMethod(...) early if you intend to make POST requests.
fasthttp sending its default “User-Agent”
Interestingly it looks like the unexpected “fasthttp” user-agent is a bug that is also caused by caching. If we look at the
RequestHeader.AppendBytes(...), which builds the header args, it performs the following check
1 2 3 4 5 6 7 8 9 10 11 12 13 14
We can see, on line 13, that the
userAgent value is taken from the field
RequestHeader.userAgent, we could quickly confirm that our preferred header value “Test-Agent” was held inside a field
RequestHeader.h but is completely missed by the call to
h.parseRawHeaders which looks inside
RequestHeader.rawHeaders. This complex arrangement of headers and various cached values makes interacting with headers a true minefield of unexpected behaviour.
Should You Use Fasthttp
It’s a difficult question to answer.
Fasthttp does reduce allocations and I have no doubt it will bring significant benefits to some systems, particularly those performing high volume HTTP requests and not much else. Garbage collection is not free, and
fasthttp could bring real performance improvements, and potentially reduce your hardware requirements. But,
fasthttp is not simple and it appears that
fasthttp has been built primarily for use on servers.
The client implementation reuses the data structures used on the server, this means, for example, that the
fasthttp.Response used by the client contains a very large amount of code which is only useful to a server. This makes understanding the codebase and debugging any problems much harder.
The high level of complexity and the likelihood that the
fasthttp client has not been extensively used in production means that you would need to expect a very large benefit to justify the adoption of
We would like to thank Valyala and other contributors for making a high performance http library available for Go. We know it is no small task.