Golang Benchmark and Testing [Step-by-Step]

In GO | Tags: #golang
Golang Benchmark and Testing [Step-by-Step]

Golang Benchmarking Code

Benchmarkingmeasures the performance of a function or program, allowing you to compare implementations and to understand the performance impact of code changes. Using that information, you can easily reveal the part of the code that needsto be rewritten to improve its performance.

Go follows certainconventions regarding benchmarking. The most important convention is that the name of a benchmark function must begin withBenchmark. After theBenchmarkword, we can put an underscore or an uppercase letter. Therefore, bothBenchmarkFunctionName()andBenchmark_functionName()are validbenchmark functions whereasBenchmarkfunctionname()is not. The same rule applies to testing functions that begin withTest. Although we are allowed to put the testing and benchmarking code on the same file with the other code, it should be avoided. By convention such functions are put in files that end with_test.go. Once the benchmarking or the testing code is correct, thego testsubcommand does all the dirty work for you, which includes scanning all*_test.gofiles for special functions, generating a proper temporarymainpackage, calling these specialfunctions, getting the results, and generating the final output.

Starting from Go 1.17, we can shuffle the execution order ofboth tests andbenchmarkswith the help of theshuffleparameter (go test -shuffle=on). Theshuffleparameter accepts a value, which is the seed for the random number generator, and can be useful when you want to replay an execution order. Its defaultvalue isoff. The logic behind that capability is that sometimes the order inwhich tests and benchmarks are executed affects their results.

There are a number of flags that are available during the invocation of the benchmarking utility. A few helpful flags for benchmarking can be found below:

  • -benchtime t : Run enough iterations of the test to take the defined t duration. Increasing this value will run more iterations ofb.N.
  • -count n : Run each test n times.
  • -benchmem : Turn on memory profiling for your test.
  • -cpu x,y,z : Specify a list ofGOMAXPROCSvalues for which the benchmarks should be executed.

Setup Lab Environment

To prepare for golang benchmarking, let us write a simple go code. We will create a directory tests and create all our codes inside this directory.

Add a file named main.go to the tests folder with the content shown below:

package main

import (
    "fmt"
    "sort"
)

func sortAndTotal(vals []int) (sorted []int, total int) {
    sorted = make([]int, len(vals))
    copy(sorted, vals)
    sort.Ints(sorted)
    for _, val := range sorted {
        total += val
        total++
    }
    return
}
func main() {
    nums := []int{100, 20, 1, 7, 84}
    sorted, total := sortAndTotal(nums)
    fmt.Println("Sorted Data:", sorted)
    fmt.Println("Total:", total)
}

ThesortAndTotalfunction contains a deliberate error that will help demonstrate the testing features in the next section.

$ go run main.go 
Sorted Data: [1 7 20 84 100]
Total: 212

Golang Unit Testing

Create Unit Test Cases

Unit tests are defined in files whose name ends with _test.go. To create a simple test, add a file named simple_test.go to the tests folder with the content shown below:

package main

import "testing"

func TestSum(t *testing.T) {
    testValues := []int{10, 20, 30}
    _, sum := sortAndTotal(testValues)
    expected := 60
    if sum != expected {
        t.Fatalf("Expected %v, Got %v", expected, sum)
    }
}

The Go standard library provides support for writing unit tests through thetestingpackage. Unit tests are expressed as functions whose name starts withTest, followed by a term that begins with an uppercase letter, such asTestSum. (The uppercase letter is important because the test tools will not recognize a function name such asTestsumas a unit test.)

Running Unit Tests

The test in the previous code calls the sumAndTotal function with a set of values and compared the result to the expected outcome using a standard Go comparison operator. If the result isn’t equal to the expected value, then the Fatalf method is called, which reports the test failure and stops any remaining statements in the unit test from being executed (although there are no remaining statements in this example).

# go test
--- FAIL: TestSum (0.00s)
    simple_test.go:10: Expected 60, Got 63
FAIL
exit status 1
FAIL    _/opt/deepak/scripts/goexamples/tests   0.040s

The output from the tests reports the error as well as the overall outcome of the test run. Let’s fix our code

 for _, val := range sorted {
        total += val
        // total++
    }
    return

Save the change and run thego testcommand, and the output will show that the test passes:

# go test
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.033s

A test file can contain multiple tests, which will be discovered and executed automatically. Below we have added a second test function to the simple_test.go file.

package main

import (
    "sort"
    "testing"
)

func TestSum(t *testing.T) {
    testValues := []int{10, 20, 30}
    _, sum := sortAndTotal(testValues)
    expected := 60
    if sum != expected {
        t.Fatalf("Expected %v, Got %v", expected, sum)
    }
}
func TestSort(t *testing.T) {
    testValues := []int{1, 279, 48, 12, 3}
    sorted, _ := sortAndTotal(testValues)
    if !sort.IntsAreSorted(sorted) {
        t.Fatalf("Unsorted data %v", sorted)
    }
}

TheTestSorttest verifies that thesortAndTotalfunction sorts data. Notice that I can rely on the features provided by the Go standard library in unit tests and use thesort.IntsAreSortedfunction to perform the test. Run thego testcommand, and you will see the following outcome:

# go test
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.042s

The go test command doesn’t report any detail by default, but more information can be generated by using -v as shown below:

# go test -v
=== RUN   TestSum
--- PASS: TestSum (0.00s)
=== RUN   TestSort
--- PASS: TestSort (0.00s)
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.046s

Golang Benchmark Testing

Create benchmarking code

Functions whose name started withBenchmark, followed by a term that begins with an uppercase letter, such as Sort, are benchmarks, whose execution is timed.Add a file named benchmark_test.go to the tests folder with the content shown below:

package main

import (
    "math/rand"
    "testing"
    "time"
)

func BenchmarkSort(b *testing.B) {
    rand.Seed(time.Now().UnixNano())
    size := 250
    data := make([]int, size)
    for i := 0; i < b.N; i++ {
        for j := 0; j < size; j++ {
            data[j] = rand.Int()
        }
        sortAndTotal(data)
    }
}

Performing benchmarks

TheBenchmarkSortfunction creates a slice with random data and passes it to thesortAndTotal function. Run the benchmark test:

# go test -bench . -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4            26616             44259 ns/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests   1.773s

The period following the-benchargument causes all of the benchmarks that thego testtool discovers to be performed. The period can be replaced with a regular expression to select specific benchmarks. By default, the unit tests are also performed, but since I introduced a deliberate error into theTestSum function, I used the -run argument to specify a value that won’t match any of the test function names in the project, with the result that only the benchmarks will be performed.

The following is an example of benchmark execution. In our example execution, we are profiling our existing TestSum benchmark twice. We’re also using four GOMAXPROCS, viewing the memory profiling for our test, and performing these requests for 2 seconds instead of the default 1-second test invocation. We can invoke our go test -bench functionality like this:

# go test -bench=. -benchtime 2s -count 2 -benchmem -cpu 4 -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4           101664             23184 ns/op            2072 B/op          2 allocs/op
BenchmarkSort-4           106664             22301 ns/op            2072 B/op          2 allocs/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests 22.304s

Abenchmarkwill run until the function returns, fails, or skips. The results of thebenchmarkare returned as a standard error once the test has completed. After the tests have completed and the results have been collated, we can make smart comparisons about the results of our benchmarks.

In our output result, we can see a couple of different bits of data being returned:

The name of the benchmark that was run, followed by the following:

  • -4: The number of GOMAXPROCS that were used to execute the tests.
  • 101664: The number of times our loop ran to gather the necessary data.
  • 23184 ns/op: The speed per loop during our test.
  • PASS: Indicates the end state of our benchmark run.
  • The final line of the test, with a compilation of the end state of our test run (ok), the path that we ran the test on, and the total time of our test run.

Removing Setup from the Benchmark

For each iteration of the for loop, the BenchmarkSort function has to generate random data, and the time taken to produce this data is included in the benchmark results. We can use following methods for timing control:

  • StopTimer() : This method stops the timer.
  • StartTimer() : This method starts the timer.
  • ResetTimer() : This method resets the timer.

TheResetTimermethod is useful when a benchmark requires some initial setup, and the other methods are useful when there is overhead associated with each benchmarked activity.

package main

import (
    "math/rand"
    "testing"
    "time"
)

func BenchmarkSort(b *testing.B) {
    rand.Seed(time.Now().UnixNano())
    size := 250
    data := make([]int, size)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        b.StopTimer()
        for j := 0; j < size; j++ {
            data[j] = rand.Int()
        }
        b.StartTimer()
        sortAndTotal(data)
    }
}

The timer is reset after the random seed is set and the slice has been initialized. Within theforloop, theStopTimermethod is used to stop the timer before the slice is populated with random data, and theStartTimermethod is used to start the timer before thesortAndTotalfunction is called.

# go test -bench . -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4            52544             21956 ns/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests   5.454s

Excluding the work required to prepare for the benchmark has produced a more accurate assessment of the time taken to execute thesortAndTotalfunction.

Summary

In this tutorial we learned about the golang benchmark and testing features.

While you are running the benchmarks in this book, be sure to remember that benchmarks aren’t the be-all and end-all for performance results. Benchmarking has both positives and drawbacks:

The positives of benchmarking are as follows:

  • Surfaces potential problems before they become unwieldy
  • Helps developers have a deeper understanding of their code
  • Can identify potential bottlenecks in the design and data structures and algorithms stages

The drawbacks of benchmarking are as follows:

  • Needs to be completed on a given cadence for meaningful results
  • Data collation can be difficult
  • Does not always yield a meaningful result for the problem at hand
Deepak Prasad

Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels across development, DevOps, networking, and security, delivering robust and efficient solutions for diverse projects.

View all posts →

Related Articles