Skip to content

Latest commit

 

History

History
209 lines (156 loc) · 5.85 KB

tests.md

File metadata and controls

209 lines (156 loc) · 5.85 KB

Testing

We have two main types of tests:

  • Golden File tests which run a StandardOperator on an input object, generate a manifest, and compare it to expected output.
  • E2E tests which bring up a real cluster, invoke the operators and can then check that the code is performing as expected.

We believe this combination strikes a good balance between coverage and making it easy to write tests.

The kubebuilder tests, which bring up an embedded apiserver and perform some basic testing, feel unnecessary with the other two approaches.

Golden File Tests

Background

The bulk of these tests are implemented in pkg/tests via the StandardValidator. The test scans a tests directory for *.in.yaml files, runs the operator for each of them, and compares the output to *.out.yaml files.

It is also possible to provide *.side_in.yaml file that contains resources that should exist before the resource defined in *.in.yaml is applied. These resources are going to be created before running a test, and cleaned up upon finishing. They will be accessible via Get and List.

We perform simple comparison-of-output tests, any changes to the output are treated as test failures, and printed as a diff. This means that the output is materialized and checked in to the repo; this proves to be very handy for understanding the impact of a change.

There's also a helpful "cheat" function, which rewrite the output when you run the tests locally - set the HACK_AUTOFIX_EXPECTED_OUTPUT env var to a non-empty string. This is useful when you have a big set of changes; it's just as easy to review the changes yourself in the diff and there's not a ton of value in typing them out.

This is the approach that we use in kops, and it works well - particularly with the env-var cheat code.

Usage

  1. Remove the autogenerated tests

    cd <your operator>/
find . -name "*test.go" -delete

  2. Create a new golden file test for your controller in pkg/controller/{{operator}}/{{operator}}_controller_test.go

     package {{operator}}
 
 import (
   "testing"
 
   api "{{operator}}/pkg/apis/addons/v1alpha1"
   "sigs.k8s.io/controller-runtime/pkg/tests"
 )
 
 func TestController(t *testing.T) {
 v := golden.NewValidator(t, api.SchemeBuilder)
 dr := &{{operator}}Reconciler{
 	Client: v.Client(),
 }
 err := dr.setupReconciler(v.Manager())
 if err != nil {
 	t.Fatalf("creating reconciler: %v", err)
 }

 v.Validate(dr.Reconciler)
 }

  3. Create a basic input to the test based on the autogenerated CR instance. This should be updated if you have any required fields

    mkdir --p pkg/controller/{{operator}}/tests
cp k8s/resources/addons_v1alpha1_{{operator}}.yaml pkg/controller/{{operator}}/tests/simple.in.yaml

  4. Generate the test output

    cd pkg/controller/{{operator}}/tests
touch tests/simple.out.yaml
HACK_AUTOFIX_EXPECTED_OUTPUT=1 go test ./...

  5. Verify the output is reproducible

    go test ./...

  6. Update the BUILD.bazel file your controller package to pull in the channel and test data

    filegroup(
    name = "testdata",
    srcs = glob(["tests/**"]),
)

go_test(
    name = "go_default_test",
    srcs = ["{{operator}}_controller_test.go"],
    data = [
        "testdata",
        "//{{operator}}-operator/channels",
    ],
   // remainder unchanged, eg embeded, deps, ..
)

  7. Verify bazel can run the test:

    bazel test //{{operator}}-operator/...

E2E tests

Background

End to End tests spin up a Kubernetes cluster with no default addon management via a modified kube-up.sh, deploy each operator to the cluster, create a CR for each operator, and run each operators validation.

An example is to deploy KubeProxy, create a KubeProxy CR, then validate the KubeProxy DaemonSet is deployed.

Setup

Your operator needs to provide a set of manifests in a the following structure:

foo-operator/
  k8s/
    crds/ - FooCrd lives in here
    operators/ - All the manifests to deploy the operator (including RBAC)
    resources/ - An example CR instance of Foo

This structure is not autogenerated. You can use the generated kubebuilder manifests to populate it:

cd foo-operator/
mkdir -p k8s/crds k8s/operators k8s/resources
cp config/crds/* k8s/crds
cp config/manager/* k8s/operators
cp config/samples/* k8s/resources

Adding Tests

For this example, we will be adding a test for KubeDNS. All code changes will live in smoketest.go.

  1. Create a test class that inherits from CommonAddonsTest and a constructor.

    type KubednsTest struct {
  CommonAddonTest
}
 
func NewKubednsTest(c CommonAddonTest) *KubednsTest {
  t := &KubednsTest{CommonAddonTest: c}
  t.Base = "kubedns-operator"
  return t
}

    This will deploy your operator from k8s/{crds,operators} and CR from k8s/resources/ .

  2. Create a verify function to assert your operator successfully deployed its workload:

    func (k *KubednsTest) Verify() error {
	h := k.Harness

	err := verifyReadyPods(h, defaultSystemNamespace, "kube-dns")
	if err != nil {
		return err
	}

	return nil
}

    This function will be called periodically until it returns nil (indicating success) or a timeout is reached (indicating failure).

    There are several verify* functions avaliable for your test to use. See other implementations for inspiration.

Running Tests

By default, the test will use your local kubeconfig to deploy and test the operators:

go run smoketest.go

You can create and destroy a fresh cluster with the following flags:

go run smoketest.go --cluster-up=true --cluster-down=true