This Programmer Tried to Mock an HTTP/2 Server in Go and Here's What Happened

Testing has always been a must at Adjust. Serving tens of thousands of requests per second, we can’t afford tiny oopsies caused by some pointer being null when we didn’t expect it or a good old side effect in a presumably clean function. Tests are considered first-class citizens in our code base and are mandatory for any pull request that adds or changes functionality.

Just like tests, any dependency that we bring into our projects becomes a part of our code base. Since no one likes maintaining code written by some unknown dude (do you?), we try to reduce the number of third-party packages we use to an absolute minimum and prefer writing or own tailor-made libraries.

This is why we decided to stick with the minimalistic but decent testing package that Go provides as a part of its standard library.

Every test, regardless of programming language, contains four major parts: setup, invocation, assertion, and teardown. Besides providing the framework to run tests using the go tool, testing provides only one of these things–assertion–and leaves the rest up to its user. While the invocation part is usually very specific to implementation and there is not much that could be generalized there, setup and teardown are tightly coupled and can often be split into small reusable blocks that mock external services, provide data fixtures, etc. So we’ll focus on these two parts, specifically on mocking HTTP requests to external services.

Mocking an external HTTP API in tests

A common task in our tests is to verify the interaction with an HTTP API provided by a third party. We don’t want to send all the requests we’re making in our unit tests to an actual server for several reasons:

  1. This would add an extra dependency on the network state between our CI server and the target service
  2. It would be difficult to reproduce the error responses
  3. Any API change would cause our tests to fail. Some would say that this is good rather than bad, but we believe that test failures should be induced by only our changes.
  4. We’re nice people and wouldn’t want to bother the API providers unless we really have to

Thus we came up with a fairly simple idea to mock http.Client:

testutils/client_mock.go
1
2
3
4
5
6
7
8
9
10
11
12
type ClientMock struct {
  Responses map[string][]byte
}

func (m *ClientMock) Get(url string) (resp *http.Response, err error) {
  body, ok := m.Responses[url]
  if !ok {
      // respond with HTTP 404 and return
  }

  // build an *http.Response with a copy of body and return it
}

Now we only needed to mock the expected requests with canned responses:

external_service.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
type Getter interface {
  Get(url string) (resp *http.Response, err error)
}

type ExternalServiceClient struct {
  HTTPClient Getter
}

func (c *ExternalServiceClient) Call() (result *Result, err error) {
  // ...
  httpClient := c.HTTPClient
  if httpClient == nil {
      httpClient = http.DefaultClient
  }

  resp, err := httpClient.Get(url)
  // ...
}
external_service_test.go
1
2
3
4
5
6
7
8
9
10
11
func TestExternalServiceClient_Call(t *testing.T) {
  mock := &testutil.ClientMock{
      Responses: map[string][]byte{
          "/": []byte("Hello, world!"),
      },
  }

  c := ExternalServiceClient{HTTPClient: mock}
  result, err := c.Call()
  // ...
}

This approach worked well for a while, but soon we found ourselves adding more and more functionality to testutils.ClientMock. Sometimes we’d need to add additional cookies to the response, send requests using different HTTP methods, or provide a different response depending on what was sent in the request. The mock became so complex, that we started thinking about writing tests for it.

Nobody was smitten with the idea of writing tests for tests, so we had to rethink our approach. By that time, our client mock looked almost like a limited http.Server without the transport part, so we decided to leave the honorable task of testing mocks to the Go team and came up with the following approach, which is currently used in most of our tests these days:

testutil/server_mock.go
1
2
3
4
5
6
7
8
9
10
import (
  "net/http"
  "net/http/httptest"
)

func ServerMock() (baseURL string, mux *http.ServeMux, teardownFn func()) {
  mux = http.NewServeMux()
  srv := httptest.NewServer(mux)
  return srv.URL, mux, srv.Close
}

And the code turned into something like this:

external_service.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
const DefaultBaseURL = "https://example.com"

type ExternalServiceClient struct {
  BaseURL string
  HTTPClient *http.Client
}

func (c *ExternalServiceClient) Call() (result *Result, err error) {
  // ...
  httpClient := c.HTTPClient
  if httpClient == nil {
      httpClient = http.DefaultClient
  }

  baseURL := c.BaseURL
  if baseURL == "" {
      baseURL = DefaultBaseURL
  }

  resp, err := httpClient.Get(baseURL + path)
  // ...
}

We’re now using the standard library *http.Client, and, instead of mocking its request methods, we override the host and port of the API server. This way we can redirect every HTTP request to our server mock:

external_service_test.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func TestExternalServiceClient_Call(t *testing.T) {
  baseURL, mux, teardown := testutil.ServerMock()
  defer teardown()

  var reqNum int
  mux.Handle("/", func(w http.ResponseWriter, req *http.Request) {
      reqNum++
      // inspect request

      w.Write([]byte("Hello, world!"))
  })

  c := ExternalServiceClient{BaseURL: baseURL}
  result, err := c.Call()
  // ...

  if expectedReqNum := 1; reqNum != expectedReqNum {
      t.Errorf("ExternalServiceClient.Call() expected to make %d request(s), but it sent %d instead", expectedReqNum, reqNum)
  }
}

Mocking an HTTPS server

For tests that explicitly require HTTPS, we added a similar mock that creates an instance of httptest.Server by calling httptest.StartTLSServer() instead of httptest.StartServer(), while the rest of the code is completely the same as in testutil.ServerMock():

testutil/server_mock.go
1
2
3
4
5
func TLSServerMock() (baseURL string, mux *http.ServeMux, teardownFn func()) {
  mux = http.NewServeMux()
  srv := httptest.NewTLSServer(mux)
  return srv.URL, mux, srv.Close
}

But this made http.Client complain about a bad certificate:

1
2
$ go test
 examples_test.go:33: ExternalServiceClient.Call() returned error: Get https://127.0.0.1:55060: x509: certificate signed by unknown authority

Since we did not provide any certificates at all while creating an httptest.Server instance, there should have been some default one hidden in net/http/httptest. It turned out that the Go standard library contains a self-signed certificate and a private key used by the httptest package. So we needed to make the http.Client trust this certificate:

server_mock.go
1
2
3
4
5
6
7
8
9
10
11
func ServerMock() (baseURL string, mux *http.ServeMux, cert *x509.Certificate, teardownFn func()) {
  mux = http.NewServeMux()
  srv := httptest.NewTLSServer(mux)

  cert, err := x509.ParseCertificate(srv.TLS.Certificates[0].Certificate[0])
  if err != nil {
      panic(fmt.Sprintf("failed to parst httptest.Server TLS cert: %s", err))
  }

  return srv.URL, mux, cert, srv.Close
}

If you’ve already upgraded to Go 1.9, then you don’t need x509.ParseCertificate() anymore. An instance of httptest.Server now has a Certificate() method that returns an *x509.Certificate used by this server.

All that was left now was to replace the system-default http.Client certificate pool with our own one, which held our server mock certificate:

external_service_test.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
func TestExternalServiceClient_Call(t *testing.T) {
  baseURL, mux, cert, teardown := testutil.ServerMock()
  defer teardown()

  // mock handlers

  certPool := x509.NewCertPool()
  certPool.AddCert(cert)

  httpClient := &http.Client{
      Transport: &http.Transport{
          TLSClientConfig: &tls.Config{
              RootCAs: certPool,
          },
      },
  }

  c := ExternalServiceClient{
      BaseURL:    baseURL,
      HTTPClient: httpClient,
  }

  result, err := c.Call()
  // ...
}

So what about HTTP/2?

Since Go v1.6, http.Server supports HTTP/2 out of the box, and we naturally assumed httptest.Server would, too. However, once we configured http.Client to use http2.Transport from golang.org/x/net/http2, our tests that used testutil.ServerMock() began to fail with an unexpected error:

1
2
3
$ go test
--- FAIL: TestExternalServiceClient_Call (0.00s)
  examples_test.go:29: ExternalServiceClient.Call() returned error: Get https://127.0.0.1:54607: http2: unexpected ALPN protocol ""; want "h2"

A quick note on ALPN ALPN or Application-Layer Protocol Negotiation is a TLS extension that allows parties to agree on which protocol should be handled over a secure connection. HTTP/2 uses this feature to avoid additional round trips, and, hence, TLS handshakes, by agreeing on an application protocol during the hello phase. The client provides a list of protocols it supports and the server is expected to choose one and send it back.

So unexpected ALPN protocol ""; want "h2" meant that our server did not know it now supported HTTP/2. There is a method in the http2 library to configure an existing server to support HTTP/2, but it expects an instance of http.Server as an argument, whereas we only had httptest.Server. Passing (httptest.Server).Config as an argument to http2.ConfigureServer() wouldn’t work, because httptest.Server uses Config to serve incoming connections using an already existing tls.Listener that is created when (*httptest.Server).StartTLS() gets called, and ALPN support is implemented by crypto/tls. Thus we needed a way to configure the httptest.Server listener to support "h2" as an application-level protocol.

$GOROOT/src/net/http/httptest/server.go
1
2
3
4
5
6
7
8
type Server struct {
  // ...
  // TLS is the optional TLS configuration, populated with a new config
  // after TLS is started. If set on an unstarted server before StartTLS
  // is called, existing fields are copied into the new config.
  TLS *tls.Config
  // ...
}

Looks exactly like what we’re looking for! What was left was to apply the same configuration changes as http2.ConfigureServer() does and we’d have a nicely working HTTP/2 mock using Go standard library only:

server_mock.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
func TLSServerMock() (baseURL string, mux *http.ServeMux, cert *x509.Certificate, teardownFn func()) {
  mux = http.NewServeMux()

  srv := httptest.NewUnstartedServer(mux)
  srv.TLS = &tls.Config{
      CipherSuites: []uint16{tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256},
      NextProtos:   []string{http2.NextProtoTLS},
  }
  srv.StartTLS()

  cert, err := x509.ParseCertificate(srv.TLS.Certificates[0].Certificate[0])
  if err != nil {
      panic(err)
  }

  return srv.URL, mux, cert, srv.Close
}

Here, http2.NextProtoTLS is a constant for the "h2" string we were looking for and tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is a cipher suite required by the HTTP/2 specification.

Conclusion

Instead of mocking http.Client, mock the server it talks to. The Go standard library offers a very convenient net/http/httptest package that allows you to spawn an HTTP server with only a few lines of code, which can be easily configured to handle HTTP/2 requests.

Comments