Slide 1
Slide 1 text
ADVANCED TESTING WITH GO
Slide 2
Slide 2 text
Mitchell Hashimoto
@mitchellh
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Primary language for ~4 years
Deployed by millions, plus significantly in enterprise
Distributed systems (Consul, Serf, Nomad, etc.)
Extreme performance (Consul, Nomad)
Security (Vault)
Correctness (Terraform, but also Consul, Nomad, Vault)
HASHICORP GO
Slide 7
Slide 7 text
TWO PARTS OF TESTING
Slide 8
Slide 8 text
TEST METHODOLOGY
WRITING TESTABLE CODE
Slide 9
Slide 9 text
TEST METHODOLOGY
WRITING TESTABLE CODE
SLIDE STYLE!
Slide 10
Slide 10 text
✦ Methods to test specific cases
✦ Techniques to write beRer tests
✦ A lot more to tesSng than “assert(func() == expected)”
TEST METHODOLOGY
Slide 11
Slide 11 text
✦ How to write code that can be tested well and easily
✦ Just as important as wriSng good tests is
wriSng code that can be tested well
✦ Many developers that tell me “this can’t be tested” aren’t wrong, they
just wrote the code in a way that made it so. We very rarely see cases
at HashiCorp that truly can’t be tested [well].
✦ RewriSng exisSng code to be testable is a pain, but worth it
TESTABLE CODE
Slide 12
Slide 12 text
TABLE DRIVEN TESTS
Slide 13
Slide 13 text
TABLE DRIVEN TESTS
func TestAdd(t *testing.T) {
cases := []struct{ A, B, Expected int }{
{ 1, 1, 2 },
{ 1, -1, 0 },
{ 1, 0, 1 },
{ 0, 0, 0 },
}
for _, tc := range cases {
actual := tc.A + tc.B
if actual != expected {
t.Errorf(
“%d + %d = %d, expected %d”,
tc.A, tc.B, actual, tc.Expected)
}
}
}
Slide 14
Slide 14 text
✦ Low overhead to add new test cases
✦ Makes tesSng exhausSve scenarios simple
✦ Makes reproducing reported issues simple
✦ Do this paRern a lot
✦ Follow paRern even for single cases, if its possible to grow
TABLE DRIVEN TESTS
Slide 15
Slide 15 text
TABLE DRIVEN TESTS
func TestAdd(t *testing.T) {
cases := map[string]struct{ A, B, Expected int }{
“foo”: { 1, 1, 2 },
“bar”: { 1, -1, 0 },
}
for k, tc := range cases {
actual := tc.A + tc.B
if actual != expected {
t.Errorf(
“%s: %d + %d = %d, expected %d”,
k, tc.A, tc.B, actual, tc.Expected)
}
}
}
(CONSIDER NAMING CASES)
Slide 16
Slide 16 text
TEST FIXTURES
Slide 17
Slide 17 text
TEST FIXTURES
func TestAdd(t *testing.T) {
data := filepath.Join(“test-fixtures”, “add_data.json”)
// … Do something with data
}
Slide 18
Slide 18 text
✦ “go test” sets pwd as package directory
✦ Use relaSve path “test-fixtures” directory as a place to store test data
✦ Very useful for loading config, model data, binary data, etc.
TEST FIXTURES
Slide 19
Slide 19 text
GOLDEN FILES
(ALSO TEST FLAGS)
Slide 20
Slide 20 text
GOLDEN FILES
var update = flag.Bool(“update”, false, “update golden files”)
func TestAdd(t *testing.T) {
// … table (probably!)
for _, tc := range cases {
actual := doSomething(tc)
golden := filepath.Join(“test-fixtures”, tc.Name+”.golden”)
if *update {
ioutil.WriteFile(golden, actual, 0644)
}
expected, _ := ioutil.ReadFile(golden)
if !bytes.Equal(actual, expected) {
// FAIL!
}
}
}
Slide 21
Slide 21 text
GOLDEN FILES
$ go test
…
$ go test -update
…
Slide 22
Slide 22 text
✦ Test complex output without manually hardcoding it
✦ Human eyeball the generated golden data. If it is correct, commit it.
✦ Very scalable way to test complex structures (write a String() method)
GOLDEN FILES
Slide 23
Slide 23 text
GLOBAL STATE
Slide 24
Slide 24 text
✦ Avoid it as much as possible.
✦ Instead of global state, try to make whatever is global a configuraSon
opSon using global state as the default, allowing tests to modify it.
✦ If necessary, make global state a var so it can be modified. This is a last
case scenario, though.
GLOBAL STATE
Slide 25
Slide 25 text
GLOBAL STATE
// Not good on its own
const port = 1000
// Better
var port = 1000
// Best
const defaultPort = 1000
type ServerOpts {
Port int // default it to defaultPort somewhere
}
Slide 26
Slide 26 text
TEST HELPERS
Slide 27
Slide 27 text
TEST HELPERS
func testTempFile(t *testing.T) string {
tf, err := ioutil.TempFile(“”, “test”)
if err != nil {
t.Fatalf(“err: %s”, err)
}
tf.Close()
return tf.Name()
}
Slide 28
Slide 28 text
✦ Never return errors. Pass in *tesSng.T and fail.
✦ By not returning errors, usage is much precer
since error checking is gone.
✦ Used to make tests clear on what they’re tesSng vs what is boilerplate
TEST HELPERS
Slide 29
Slide 29 text
TEST HELPERS
func testTempFile(t *testing.T) (string, func()) {
tf, err := ioutil.TempFile(“”, “test”)
if err != nil {
t.Fatalf(“err: %s”, err)
}
tf.Close()
return tf.Name(), func() { os.Remove(tf.Name()) }
}
func TestThing(t *testing.T) {
tf, tfclose := testTempFile(t)
defer tfclose()
}
Slide 30
Slide 30 text
TEST HELPERS
func testChdir(t *testing.T, dir string) func() {
old, err := os.Getwd()
if err != nil {
t.Fatalf(“err: %s”, err)
}
if err := os.Chdir(dir); err != nil {
t.Fatalf(“err: %s”, err)
}
return func() { os.Chdir(old) }
}
func TestThing(t *testing.T) {
defer testChdir(t, “/other”)()
// …
}
Slide 31
Slide 31 text
✦ Returning a func() for cleanup is an elegant way to hide that
✦ The func() is a closure that can have access to *tesSng.T to also fail
✦ Example: testChdir proper setup/cleanup would be at least 10 lines
without the helper. Now avoids that in all our tests.
TEST HELPERS
Slide 32
Slide 32 text
PACKAGE/FUNCTIONS
Slide 33
Slide 33 text
✦ Break down funcSonality into packages/funcSons judiciously
✦ NOTE: Don’t overdo it. Do it where it makes sense.
✦ Doing this correctly will aid tesSng while also improving organizaSon.
Over-doing it will complicate tesSng and readability.
✦ QualitaSve, but pracSce will make perfect.
PACKAGE/FUNCTIONS
Slide 34
Slide 34 text
✦ Unless the funcSon is extremely complex, we try to test only the
exported funcSons, the exported API.
✦ We treat unexported funcSons/structs as implementaSon details:
they are a means to an end. As long as we test the end and it behaves
within spec, the means don’t maRer.
✦ Some people take this too far and choose to only integraSon/
acceptance test, the ulSmate “test the end, ignore the means.” We
disagree with this approach.
PACKAGE/FUNCTIONS
Slide 35
Slide 35 text
NETWORKING
Slide 36
Slide 36 text
✦ TesSng networking? Make a real network connecSon.
✦ Don’t mock `net.Conn`, no point.
NETWORKING
Slide 37
Slide 37 text
NETWORKING
// Error checking omitted for brevity
func TestConn(t *testing.T) (client, server net.Conn) {
ln, err := net.Listen(“tcp”, “127.0.0.1:0”)
var server net.Conn
go func() {
defer ln.Close()
server, err = ln.Accept()
}()
client, err := net.Dial(“tcp”, ln.Addr().String())
return client, server
}
Slide 38
Slide 38 text
✦ That was a one-connecSon example. Easy to make an N-connecSon.
✦ Easy to test any protocol.
✦ Easy to return the listener as well.
✦ Easy to test IPv6 if needed.
✦ Why ever mock net.Conn? (Rhetorical, for readers)
NETWORKING
Slide 39
Slide 39 text
CONFIGURABILITY
Slide 40
Slide 40 text
✦ Unconfigurable behavior is onen a point of difficulty for tests.
✦ Example: ports, Smeouts, paths
✦ Over-parameterize structs to allow tests to fine-tune their behavior
✦ It is okay to make these configuraSons unexported so only tests can
set them.
CONFIGURABILITY
Slide 41
Slide 41 text
CONFIGURABILITY
// Do this, even if cache path and port are always the same
// in practice. For testing, it lets us be more careful.
type ServerOpts struct {
CachePath string
Port int
}
Slide 42
Slide 42 text
SUBPROCESSING
Slide 43
Slide 43 text
✦ Subprocessing is typical a point of difficult-to-test behavior.
✦ Two opSons:
1. Actually do the subprocess
2. Mock the output or behavior
SUBPROCESSING
Slide 44
Slide 44 text
✦ Actually execuSng the subprocess is nice
✦ Guard the test for the existence of the binary
✦ Make sure side effects don’t affect any other test
SUBPROCESSING: REAL
Slide 45
Slide 45 text
SUBPROCESSING: REAL
var testHasGit bool
func init() {
if _, err := exec.LookPath("git"); err == nil {
testHasGit = true
}
}
func TestGitGetter(t *testing.T) {
if !testHasGit {
t.Log("git not found, skipping")
t.Skip()
}
// …
}
Slide 46
Slide 46 text
✦ You s2ll actually execute, but you’re execuSng a mock!
✦ Make the *exec.Cmd configurable, pass in a custom one
✦ Found this in the stdlib, it is how they test os/exec!
✦ How HashiCorp tests go-plugin and more
SUBPROCESSING: MOCK
Slide 47
Slide 47 text
SUBPROCESSING: MOCK
func helperProcess(s ...string) *exec.Cmd {
cs := []string{"-test.run=TestHelperProcess", "--"}
cs = append(cs, s...)
env := []string{
"GO_WANT_HELPER_PROCESS=1",
}
cmd := exec.Command(os.Args[0], cs...)
cmd.Env = append(env, os.Environ()...)
return cmd
}
GET THE *EXEC.CMD
Slide 48
Slide 48 text
SUBPROCESSING: MOCK
func TestHelperProcess(*testing.T) {
if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
return
}
defer os.Exit(0)
args := os.Args
for len(args) > 0 {
if args[0] == "--" {
args = args[1:]
break
}
args = args[1:]
}
WHAT IT EXECUTES
Slide 49
Slide 49 text
SUBPROCESSING: MOCK
…
cmd, args := args[0], args[1:]
switch cmd {
case “foo”:
// …
WHAT IT EXECUTES
Slide 50
Slide 50 text
INTERFACES
Slide 51
Slide 51 text
✦ Interfaces are mocking points.
✦ Behavior can be defined regardless of implementaSon and exposed
via custom framework or tesSng.go (covered elsewhere)
✦ Similar to package/funcSons: do this judiciously, but overdoing it will
complicate readability.
INTERFACES
Slide 52
Slide 52 text
TESTING AS A PUBLIC API
Slide 53
Slide 53 text
✦ Newer HashiCorp projects have adopted the pracSce of making a
“tesSng.go” or “tesSng_*.go” files.
✦ These are exported APIs for the sole purpose of providing mocks, test
harnesses, helpers, etc.
✦ Allows other packages to test using our package without reinvenSng
the components needed to meaningful use our package in a test.
TESTING AS A PUBLIC API
Slide 54
Slide 54 text
✦ Example: config file parser
✦ TestConfig(t) => Returns a valid, complete configuraSon for tests
✦ TestConfigInvalid(t) => Returns an invalid configuraSon
TESTING AS A PUBLIC API
Slide 55
Slide 55 text
✦ Example: API server
✦ TestServer(t) (net.Addr, io.Closer) => Returns a fully started in-
memory server (address to connect to) and a closer to close it.
TESTING AS A PUBLIC API
Slide 56
Slide 56 text
✦ Example: interface for downloading files
✦ TestDownloader(t, Downloader) => Tests all the properSes a
downloader should have.
✦ struct DownloaderMock{} => Implements Downloder as a mock,
allowing recording and replaying of calls.
TESTING AS A PUBLIC API
Slide 57
Slide 57 text
CUSTOM FRAMEWORKS
Slide 58
Slide 58 text
✦ `go test` is an incredible workflow tool
✦ Complex, pluggable systems? Write a custom framework within
`go test`, rather than a separate test harness.
✦ Example: Terraform providers, Vault backends, Nomad schedulers
CUSTOM FRAMEWORKS
Slide 59
Slide 59 text
CUSTOM FRAMEWORKS
// Example from Vault
func TestBackend_basic(t *testing.T) {
b, _ := Factory(logical.TestBackendConfig())
logicaltest.Test(t, logicaltest.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Backend: b,
Steps: []logicaltest.TestStep{
testAccStepConfig(t, false),
testAccStepRole(t),
testAccStepReadCreds(t, b, "web"),
testAccStepConfig(t,false),
testAccStepRole(t),
testAccStepReadCreds(t, b, "web"),
},
})
}
Slide 60
Slide 60 text
✦ “logicaltest.Test” is just a custom harness doing repeated setup/
teardown, asserSons, etc.
✦ Other examples: Terraform provider acceptance tests
✦ We can sSll use `go test` to run them
CUSTOM FRAMEWORKS
Slide 61
Slide 61 text
TIMING-DEPENDENT TESTS
Slide 62
Slide 62 text
TIMING-DEPENDENT TESTS
func TestThing(t *testing.T) {
// …
select {
case <-thingHappened:
case <-time.After(timeout):
t.Fatal(“timeout”)
}
}
Slide 63
Slide 63 text
✦ We don’t use “fake Sme”
✦ We just have a mulSplier available that we can
set to increase Smeouts
✦ Not perfect, but not as intrusive as fake Sme. SSll, fake Sme could be
beRer, but we haven’t found an effecSve way to use it yet.
TIMING-DEPENDENT TESTS
Slide 64
Slide 64 text
TIMING-DEPENDENT TESTS
func TestThing(t *testing.T) {
// …
timeout := 3 * time.Minute * timeMultiplier
select {
case <-thingHappened:
case <-time.After(timeout):
t.Fatal(“timeout”)
}
}
Slide 65
Slide 65 text
PARALLELIZATION
Slide 66
Slide 66 text
TEST HELPERS
func TestThing(t *testing.T) {
t.Parallel()
}
Slide 67
Slide 67 text
✦ Don’t do it. Run mulSple processes.
✦ Makes test failures uncertain: is it due to pure logic but, or race?
✦ OR: Run tests both with `-parallel=1` and `-parallel=N`
✦ We’ve preferred to just not use parallelizaSon. We use mulSple
processes and unit tests specifically wriRen to test for races.
PARALLELIZATION
Slide 68
Slide 68 text
hashicorp
hBps:/
/hashicorp.com
QUESTIONS?
THANK YOU!