FastHTTP Client

At adjust we recently tried to replace the Go standard http library with fasthttp. Fasthttp is a low allocation, high performance HTTP library, in synthetic benchmarks the client shows a 10x performance improvement and in real systems the server has been reported to provide a 3x speedup. The service we wanted to improve makes a very large number of HTTP requests and so we were very interested in using the fasthttp client.

In the course of making the switch we encountered a number of difficulties. First the fasthttp library presents a very different interface to the programmer which must be adjusted to. Second there were a number of quirks in the implementation which made progress rather slow.

Making a simple request

To begin with we would like to learn how to perform a simple HTTP request using the fasthttp client. Below is a very simple request using the Go standard library, error handling has been omitted for brevity.

For all the code snippets below the test server writes the request’s “User-Agent” header value and body into the response on separate lines. We write the actual output of the snippet in comments beneath each print statement.

1
2
3
4
5
6
7
8
9
10
11
func doRequest(url string) {
        req, _ := http.NewRequest("GET", url, nil)

        client := &http.Client{}
        resp, _ := client.Do(req)

        bodyBytes, _ := ioutil.ReadAll(resp.Body)
        println(string(bodyBytes))
        // User-Agent: Go-http-client/1.1
        // Body:
}

A fasthttp request can be written, also without error handling, very similarly:

1
2
3
4
5
6
7
8
9
10
11
12
13
func doRequest(url string) {
        req := fasthttp.AcquireRequest()
        req.SetRequestURI(url)

        resp := fasthttp.AcquireResponse()
        client := &fasthttp.Client{}
        client.Do(req, resp)

        bodyBytes := resp.Body()
        println(string(bodyBytes))
        // User-Agent: fasthttp
        // Body:
}

Getting the response body

The body of an http.Response is exposed as an exported io.ReadCloser field. The body of a fasthttp.Response is exposed via the Body() method call which returns a []byte. The implication of this is that the entire body must be read and a sufficiently large []byte allocated before the body can be processed. This is a surprising feature of a library which prioritises performance and low memory allocations.

One curious aspect of the Body() method is that it returns no error, in contrast to reading from an io.ReadCloser. It would be interesting to see how that method is implemented to get a better idea of how fasthttp works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
type Response struct {
//...
        bodyStream io.Reader
        body *bytebufferpool.ByteBuffer
//...
}

// Body returns response body.
func (resp *Response) Body() []byte {
        if resp.bodyStream != nil {
                bodyBuf := resp.bodyBuffer()
                bodyBuf.Reset()
                _, err := copyZeroAlloc(bodyBuf, resp.bodyStream)
                resp.closeBodyStream()
                if err != nil {
                        bodyBuf.SetString(err.Error())
                }
        }
        return resp.bodyBytes()
}

func (resp *Response) bodyBuffer() *bytebufferpool.ByteBuffer {
        if resp.body == nil {
                resp.body = responseBodyPool.Get()
        }
        return resp.body
}

func (resp *Response) bodyBytes() []byte {
        if resp.body == nil {
                return nil
        }
        return resp.body.B
}

The Body() method operates on two unexported fields body and bodyStream. It first checks if bodyStream is non-nil, and if it is, reads from the bodyStream into the body field. Finally the contents of the body field are returned to the caller.

This is pleasantly straightforward, but there is one odd wrinkle, this method will silently eat errors.

Looking at line 15 in the example above we can see that any errors encountered while reading from bodyStream are written into the body field and the original error is not returned. An error could occur, but we would never find out about it. Lets look further into our simple HTTP request example to see how the Body() method would actually execute.

If we trace the execution of our simple request above we find the following execution path:

1
2
3
4
5
(*Client) Do(...)                   fasthttp/client.go:356
(*HostClient) Do(...)               fasthttp/client.go:969
(*HostClient) do(...)               fasthttp/client.go:995
(*HostClient) doNonNilReqResp(...)  fasthttp/client.go:1011
(*Response) ReadLimitBody(...)      fasthttp/http.go:966

Inside ReadLimitBody(...) we find this critical piece of code

1
2
3
4
5
6
7
    bodyBuf := resp.bodyBuffer()
    bodyBuf.Reset()
    bodyBuf.B, err = readBody(r, resp.Header.ContentLength(), maxBodySize, bodyBuf.B)
    if err != nil {
            resp.Reset()
            return err
    }

We can see, on line 3, that the call to readBody(...) sets the bytes bodyBuf.B to be the result of reading from the connection. So the stream reader field will be nil. We can see that errors are being returned from the readBody(...) method call. That’s good, but we have only covered one simple case. From further analysis I do believe that errors are not swallowed by the fasthttp client, but I am not certain. There is a potential execution path which results in errors being silently swallowed.

Making POST requests

Our existing application performs both GET and POST requests. We ran into a small problem making POST requests. We will start with a simple POST example using fasthttp. Here we set our method to POST and fill in the body with some form-encoded values. Now we see both the “User-Agent” and the non-empty request body in the response.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func doRequest(url string) {
        req := fasthttp.AcquireRequest()
        req.SetRequestURI(url)
        req.Header.SetMethod("POST")
        req.SetBodyString("p=q")

        resp := fasthttp.AcquireResponse()
        client := &fasthttp.Client{}
        client.Do(req, resp)

        bodyBytes := resp.Body()
        println(string(bodyBytes))
        // User-Agent: fasthttp
        // Body: p=q
}

Setting your “User-Agent”

Next we want to set our “User-Agent” header manually, but there is a small problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func doRequest(url string) {
        req := fasthttp.AcquireRequest()
        req.SetRequestURI(url)
        req.Header.Add("User-Agent", "Test-Agent")
        req.Header.SetMethod("POST")
        req.SetBodyString("p=q")

        resp := fasthttp.AcquireResponse()
        client := &fasthttp.Client{}
        client.Do(req, resp)

        bodyBytes := resp.Body()
        println(string(bodyBytes))
        // User-Agent: fasthttp
        // Body: p=q
}

While the standard library http.Client does provide a default “User-Agent” header value, this value is overridden when any other value is provided. Fasthttp is still sending it’s default “fasthttp” and our “Test-Agent” value is not being picked up.

We wanted to get a better look at the headers that were being set, so we added a single debug line println(req.Header.String()). Now we can no longer ignore errors in our code, because that innocent looking req.Header.String() causes client.Do(...) to fail.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
func doRequest(url string) {
        req := fasthttp.AcquireRequest()
        req.SetRequestURI(url)
        req.Header.Add("User-Agent", "Test-Agent")

        println(req.Header.String())
        // GET http://127.0.0.1:61765 HTTP/1.1
        // User-Agent: fasthttp
        // User-Agent: Test-Agent

        req.Header.SetMethod("POST")
        req.SetBodyString("p=q")

        resp := fasthttp.AcquireResponse()
        client := &fasthttp.Client{}
        if err := client.Do(req, resp); err != nil {
                println("Error:", err.Error())
        } else {
                bodyBytes := resp.Body()
                println(string(bodyBytes))
        }
        // Error: non-zero body for non-POST request. body="p=q"
}

When we print the request headers we get to see the preloaded “User-Agent: fasthttp” header value is still stored, and in particular ahead of our “Test-Agent” value. This certainly explains why we aren’t seeing our value. We will look into why this is happening after we deal with the request error.

Why do our requests return an error?

After adding a simple println statement we now get the error “Error: non-zero body for non-POST request. body=”p=q”“. The client now seems to believe that our request is not a POST. The critical call path here is

1
2
3
4
5
fasthttp.RequestHeader.String()       fasthttp/header.go:1408
fasthttp.RequestHeader.Header()       fasthttp/header.go:1402
fasthttp.RequestHeader.AppendBytes()  fasthttp/header.go:1414
fasthttp.RequestHeader.noBody()       fasthttp/header.go:1515
fasthttp.RequestHeader.IsGet()        fasthttp/header.go:500

We can look into the IsGet() method to see some interesting caching behaviour.

1
2
3
4
5
6
7
8
// IsGet returns true if request method is GET.
func (h *RequestHeader) IsGet() bool {
        // Optimize fast path for GET requests.
        if !h.isGet {
                h.isGet = bytes.Equal(h.Method(), strGet)
        }
        return h.isGet
}

The method IsGet() reads the RequestHeader.method field and sets the RequestHeader.isGet cache field, to speed up future method calls. Unfortunately at this point we haven’t set our method and in the absence of any value it defaults to GET. So RequestHeader.isGet is set to true, which causes future calls to IsGet() to return true regardless of the value the RequestHeader.method field. Critically this method is also called inside HostClient.doNonNilReqResp(...) to test whether the request should have an empty body or not, causing the error we see above.

It’s worth noting that the call path contains 4 exported methods, any one of which would create the same confusing behaviour. You must be very careful to call req.Header.SetMethod(...) early if you intend to make POST requests.

Why is fasthttp sending its default “User-Agent”

Interestingly it looks like the unexpected “fasthttp” user-agent is a bug that is also caused by caching. If we look at the RequestHeader.AppendBytes(...), which builds the header args, it performs the following check

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func (h *RequestHeader) AppendBytes(dst []byte) []byte {
//...
        userAgent := h.UserAgent()
        if len(userAgent) == 0 {
                userAgent = defaultUserAgent
        }
        dst = appendHeaderLine(dst, strUserAgent, userAgent)
//...
}

func (h *RequestHeader) UserAgent() []byte {
        h.parseRawHeaders()
        return h.userAgent
}

We can see, on line 13, that the userAgent value is taken from the field RequestHeader.userAgent, we could quickly confirm that our preferred header value “Test-Agent” was held inside a field RequestHeader.h but is completely missed by the call to h.parseRawHeaders which looks inside RequestHeader.rawHeaders. This complex arrangement of headers and various cached values makes interacting with headers a true minefield of unexpected behaviour.

Should You Use Fasthttp

It’s a difficult question to answer. Fasthttp does reduce allocations and I have no doubt it will bring significant benefits to some systems, particularly those performing high volume HTTP requests and not much else. Garbage collection is not free, and fasthttp could bring real performance improvements, and potentially reduce your hardware requirements. But, fasthttp is not simple and it appears that fasthttp has been built primarily for use on servers.

The client implementation reuses the data structures used on the server, this means, for example, that the fasthttp.Response used by the client contains a very large amount of code which is only useful to a server. This makes understanding the codebase and debugging any problems much harder.

The high level of complexity and the likelihood that the fasthttp client has not been extensively used in production means that you would need to expect a very large benefit to justify the adoption of fasthttp today.

Thanks

We would like to thank Valyala and other contributors for making a high performance http library available for Go. We know it is no small task.

Comments