Testing has always been a must at Adjust. Serving tens of thousands of requests per second, we can’t afford tiny oopsies caused by some pointer being null
when we didn’t expect it or a good old side effect in a presumably clean function. Tests are considered first-class citizens in our code base and are mandatory for any pull request that adds or changes functionality.
Just like tests, any dependency that we bring into our projects becomes a part of our code base. Since no one likes maintaining code written by some unknown dude (do you?), we try to reduce the number of third-party packages we use to an absolute minimum and prefer writing or own tailor-made libraries.
This is why we decided to stick with the minimalistic but decent testing
package that Go provides as a part of its standard library.
Every test, regardless of programming language, contains four major parts: setup, invocation, assertion, and teardown. Besides providing the framework to run tests using the go
tool, testing
provides only one of these things–assertion–and leaves the rest up to its user. While the invocation part is usually very specific to implementation and there is not much that could be generalized there, setup and teardown are tightly coupled and can often be split into small reusable blocks that mock external services, provide data fixtures, etc. So we’ll focus on these two parts, specifically on mocking HTTP requests to external services.
Mocking an external HTTP API in tests
A common task in our tests is to verify the interaction with an HTTP API provided by a third party. We don’t want to send all the requests we’re making in our unit tests to an actual server for several reasons:
- This would add an extra dependency on the network state between our CI server and the target service
- It would be difficult to reproduce the error responses
- Any API change would cause our tests to fail. Some would say that this is good rather than bad, but we believe that test failures should be induced by only our changes.
- We’re nice people and wouldn’t want to bother the API providers unless we really have to
Thus we came up with a fairly simple idea to mock http.Client
:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Now we only needed to mock the expected requests with canned responses:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
1 2 3 4 5 6 7 8 9 10 11 |
|
This approach worked well for a while, but soon we found ourselves adding more and more functionality to testutils.ClientMock
. Sometimes we’d need to add additional cookies to the response, send requests using different HTTP methods, or provide a different response depending on what was sent in the request. The mock became so complex, that we started thinking about writing tests for it.
Nobody was smitten with the idea of writing tests for tests, so we had to rethink our approach. By that time, our client mock looked almost like a limited http.Server
without the transport part, so we decided to leave the honorable task of testing mocks to the Go team and came up with the following approach, which is currently used in most of our tests these days:
1 2 3 4 5 6 7 8 9 10 |
|
And the code turned into something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
We’re now using the standard library *http.Client
, and, instead of mocking its request methods, we override the host and port of the API server. This way we can redirect every HTTP request to our server mock:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
Mocking an HTTPS server
For tests that explicitly require HTTPS, we added a similar mock that creates an instance of httptest.Server
by calling httptest.StartTLSServer()
instead of httptest.StartServer()
, while the rest of the code is completely the same as in testutil.ServerMock()
:
1 2 3 4 5 |
|
But this made http.Client
complain about a bad certificate:
1 2 |
|
Since we did not provide any certificates at all while creating an httptest.Server
instance, there should have been some default one hidden in net/http/httptest
. It turned out that the Go standard library contains a self-signed certificate and a private key used by the httptest
package. So we needed to make the http.Client
trust this certificate:
1 2 3 4 5 6 7 8 9 10 11 |
|
If you’ve already upgraded to Go 1.9, then you don’t need x509.ParseCertificate()
anymore. An instance of httptest.Server
now has a Certificate()
method that returns an *x509.Certificate
used by this server.
All that was left now was to replace the system-default http.Client
certificate pool with our own one, which held our server mock certificate:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
So what about HTTP/2?
Since Go v1.6, http.Server
supports HTTP/2 out of the box, and we naturally assumed httptest.Server
would, too. However, once we configured http.Client
to use http2.Transport
from golang.org/x/net/http2
, our tests that used testutil.ServerMock()
began to fail with an unexpected error:
1 2 3 |
|
A quick note on ALPN ALPN or Application-Layer Protocol Negotiation is a TLS extension that allows parties to agree on which protocol should be handled over a secure connection. HTTP/2 uses this feature to avoid additional round trips, and, hence, TLS handshakes, by agreeing on an application protocol during the hello phase. The client provides a list of protocols it supports and the server is expected to choose one and send it back.
So unexpected ALPN protocol ""; want "h2"
meant that our server did not know it now supported HTTP/2. There is a method in the http2
library to configure an existing server to support HTTP/2, but it expects an instance of http.Server
as an argument, whereas we only had httptest.Server
. Passing (httptest.Server).Config
as an argument to http2.ConfigureServer()
wouldn’t work, because httptest.Server
uses Config
to serve incoming connections using an already existing tls.Listener
that is created when (*httptest.Server).StartTLS()
gets called, and ALPN support is implemented by crypto/tls
. Thus we needed a way to configure the httptest.Server
listener to support "h2"
as an application-level protocol.
1 2 3 4 5 6 7 8 |
|
Looks exactly like what we’re looking for! What was left was to apply the same configuration changes as http2.ConfigureServer()
does and we’d have a nicely working HTTP/2 mock using Go standard library only:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
Here, http2.NextProtoTLS
is a constant for the "h2"
string we were looking for and tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
is a cipher suite required by the HTTP/2 specification.
Conclusion
Instead of mocking http.Client
, mock the server it talks to. The Go standard library offers a very convenient net/http/httptest
package that allows you to spawn an HTTP server with only a few lines of code, which can be easily configured to handle HTTP/2 requests.