Running CI Locally¶
This section describes how to run CI tests on a local machine. This may be useful for expanding the CI test coverage or testing a private fix before creating a pull request.
Download and Build Kubernetes Components¶
Go Version¶
Older versions of Kubernetes do not build with newer versions of Go, specifically Kubernetes v1.16.4 doesn't build with Go version 1.13.x. If this is a version of Kubernetes that needs to be tested with, as a workaround, Go version 1.12.1 can be downloaded to a local directory and the $PATH variable updated only where kubernetes is being built.
$ go version
go version go1.13.8 linux/amd64
$ mkdir -p /home/$USER/src/golang/go1-12-1/; cd /home/$USER/src/golang/go1-12-1/
$ wget https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz
$ tar -xzf go1.12.1.linux-amd64.tar.gz
$ PATH=/home/$USER/src/golang/go1-12-1/go/bin:$GOPATH/src/k8s.io/kubernetes/_output/local/bin/linux/amd64:$PATH
Download and Build Kubernetes Components (E2E Tests, ginkgo, kubectl):¶
Determine which version of Kubernetes is currently used in CI (See
ovn-kubernetes/.github/workflows/test.yml)
and set the environmental variable K8S_VERSION to the same value. Also make sure to export a GOPATH which points to
your go directory with export GOPATH=(...).
K8S_VERSION=v1.33.1
git clone --single-branch --branch $K8S_VERSION https://github.com/kubernetes/kubernetes.git $GOPATH/src/k8s.io/kubernetes/
pushd $GOPATH/src/k8s.io/kubernetes/
make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl"
rm -rf .git
sudo cp _output/local/go/bin/e2e.test /usr/local/bin/.
sudo cp _output/local/go/bin/kubectl /usr/local/bin/kubectl-$K8S_VERSION
sudo ln -s /usr/local/bin/kubectl-$K8S_VERSION /usr/local/bin/kubectl
cp _output/local/go/bin/ginkgo /usr/local/bin/.
popd
If you have any failures during the build, verify $PATH has been updated to point to correct GO version. Also may need to change settings on some of the generated binaries. For example:
chmod +x $GOPATH/src/k8s.io/kubernetes/_output/bin/deepcopy-gen
Export environment variables¶
Before setting up KIND and before running the actual tests, export essential environment variables.
The environment variables and their values depend on the actual test scenario that you want to run.
Look at the e2e action (search for name: e2e) in ovn-kubernetes/.github/workflows/test.yml. Prior to installing kind, set the following environment variables according to your needs:
export KIND_CLUSTER_NAME=ovn
export KIND_INSTALL_INGRESS=[true|false]
export KIND_ALLOW_SYSTEM_WRITES=[true|false]
export PARALLEL=[true|false]
export JOB_NAME=(... job name ...)
export OVN_HYBRID_OVERLAY_ENABLE=[true|false]
export OVN_MULTICAST_ENABLE=[true|false]
export OVN_EMPTY_LB_EVENTS=[true|false]
export OVN_HA=[true|false]
export OVN_DISABLE_SNAT_MULTIPLE_GWS=[true|false]
export OVN_GATEWAY_MODE=["local"|"shared"]
export PLATFORM_IPV4_SUPPORT=[true|false]
export PLATFORM_IPV6_SUPPORT=[true|false]
# not required for the OVN Kind installation script, but export this already for later
export OVN_SECOND_BRIDGE=[true|false]
You can refer to a recent CI run from any pull request in https://github.com/ovn-org/ovn-kubernetes/actions to get a valid set of settings.
As an example for the control-plane-noHA-local-ipv4-snatGW-1br job, the settings are at time of this writing:
export KIND_CLUSTER_NAME=ovn
export KIND_INSTALL_INGRESS=true
export KIND_ALLOW_SYSTEM_WRITES=true
export PARALLEL=true
export JOB_NAME=control-plane-noHA-local-ipv4-snatGW-1br
export OVN_HYBRID_OVERLAY_ENABLE=true
export OVN_MULTICAST_ENABLE=true
export OVN_EMPTY_LB_EVENTS=true
export OVN_HA=false
export OVN_DISABLE_SNAT_MULTIPLE_GWS=false
export OVN_GATEWAY_MODE="local"
export PLATFORM_IPV4_SUPPORT=true
export PLATFORM_IPV6_SUPPORT=false
# not required for the OVN Kind installation script, but export this already for later
export OVN_SECOND_BRIDGE=false
KIND¶
Kubernetes in Docker (KIND) is used to deploy Kubernetes locally where a docker (or podman) container is created per Kubernetes node. The CI tests run on this Kubernetes deployment. Therefore, KIND will need to be installed locally.
Generic instructions for installing and running OVN-Kubernetes with KIND can be found at: OVN-Kubernetes KIND Setup
Make sure to set the required environment variables first (see section above). Then, deploy kind:
$ pushd contrib
$ ./kind.sh
$ popd
Run Tests¶
To run the tests locally, run a KIND deployment as described above. The E2E tests look for the kube config file in a special location, so make a copy:
cp ~/ovn.conf ~/.kube/kind-config-kind
To run the desired shard, first make sure that the necessary environment variables are exported (see section above).
Then, go to the location of your local copy of the ovn-kubernetes repository:
$ REPO=$GOPATH/src/github.com/ovn-org/ovn-kubernetes
$ cd $REPO
Running a suite of shards or control-plane tests¶
Finally, run the the shard that you want to test against (each shard can take 30+ minutes to complete)
$ pushd test
# run either
$ make shard-network
# or
$ make shard-conformance
# or
$ GITHUB_WORKSPACE="$REPO" make control-plane
# or
$ make conformance
$ popd
Running a single E2E test¶
To run a single E2E test instead, target the shard-test action, as follows:
$ cd $REPO
$ pushd test
$ make shard-test WHAT="should enforce egress policy allowing traffic to a server in a different namespace based on PodSelector and NamespaceSelector"
$ popd
As a reminder, shards use the E2E framework. The value of WHAT= will be used to modify the --focus parameter. Individual tests can be retrieved from https://github.com/kubernetes/kubernetes/tree/master/test/e2e. For network tests, one could run:
grep ginkgo.It $GOPATH/src/k8s.io/kubernetes/test/e2e/network/ -Ri
For example:
# grep ginkgo.It $GOPATH/src/k8s.io/kubernetes/test/e2e/network/ -Ri | head -1
/root/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go: ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() {
# make shard-test WHAT="should enforce policy to allow traffic from pods within server namespace based on PodSelector"
(...)
+ case "$SHARD" in
++ echo should be able to preserve UDP traffic when server pod cycles for a NodePort service
++ sed 's/ /\\s/g'
+ FOCUS='should\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sNodePort\sservice'
+ export KUBERNETES_CONFORMANCE_TEST=y
+ KUBERNETES_CONFORMANCE_TEST=y
+ export KUBE_CONTAINER_RUNTIME=remote
+ KUBE_CONTAINER_RUNTIME=remote
+ export KUBE_CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
+ KUBE_CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
+ export KUBE_CONTAINER_RUNTIME_NAME=containerd
+ KUBE_CONTAINER_RUNTIME_NAME=containerd
+ export FLAKE_ATTEMPTS=5
+ FLAKE_ATTEMPTS=5
+ export NUM_NODES=20
+ NUM_NODES=20
+ export NUM_WORKER_NODES=3
+ NUM_WORKER_NODES=3
+ ginkgo --nodes=20 '--focus=should\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sNodePort\sservice' '--skip=Networking\sIPerf\sIPv[46]|\[Feature:PerformanceDNS\]|Disruptive|DisruptionController|\[sig-apps\]\sCronJob|\[sig-storage\]|\[Feature:Federation\]|should\shave\sipv4\sand\sipv6\sinternal\snode\sip|should\shave\sipv4\sand\sipv6\snode\spodCIDRs|kube-proxy|should\sset\sTCP\sCLOSE_WAIT\stimeout|should\shave\ssession\saffinity\stimeout\swork|named\sport.+\[Feature:NetworkPolicy\]|\[Feature:SCTP\]|service.kubernetes.io/headless|should\sresolve\sconnection\sreset\sissue\s#74839|sig-api-machinery|\[Feature:NoSNAT\]|Services.+(ESIPP|cleanup\sfinalizer)|configMap\snameserver|ClusterDns\s\[Feature:Example\]|should\sset\sdefault\svalue\son\snew\sIngressClass|should\sprevent\sIngress\screation\sif\smore\sthan\s1\sIngressClass\smarked\sas\sdefault|\[Feature:Networking-IPv6\]|\[Feature:.*DualStack.*\]' --flake-attempts=5 /usr/local/bin/e2e.test -- --kubeconfig=/root/ovn.conf --provider=local --dump-logs-on-failure=false --report-dir=/root/ovn-kubernetes/test/_artifacts --disable-log-dump=true --num-nodes=3
Running Suite: Kubernetes e2e suite
(...)
• [SLOW TEST:28.091 seconds]
[sig-network] Conntrack
/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:23
should be able to preserve UDP traffic when server pod cycles for a NodePort service
/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:128
------------------------------
{"msg":"PASSED [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort service","total":-1,"completed":1,"skipped":229,"failed":0}
Aug 17 14:46:42.842: INFO: Running AfterSuite actions on all nodes
Aug 17 14:46:15.264: INFO: Running AfterSuite actions on all nodes
Aug 17 14:46:42.885: INFO: Running AfterSuite actions on node 1
Aug 17 14:46:42.885: INFO: Skipping dumping logs from cluster
Ran 1 of 5667 Specs in 30.921 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Flaked | 0 Pending | 5666 Skipped
Ginkgo ran 1 suite in 38.489055861s
Test Suite Passed
Running a control-plane test¶
All local tests are defined as control-plane tests. To run a single control-plane test, target the control-plane
action and append the WHAT=<test name> parameter, as follows:
$ cd $REPO
$ pushd test
$ make control-plane WHAT="should be able to send multicast UDP traffic between nodes"
$ popd
The value of WHAT= will be used to modify the -ginkgo.focus parameter. Individual tests can be retrieved from this
repository under test/e2e. To see a list of individual
tests, one could run:
grep -R ginkgo.It test/
For example:
# grep -R ginkgo.It . | head -1
./e2e/multicast.go: ginkgo.It("should be able to send multicast UDP traffic between nodes", func() {
# make control-plane WHAT="should be able to send multicast UDP traffic between nodes"
(...)
+ go test -timeout=0 -v . -ginkgo.v -ginkgo.focus 'should\sbe\sable\sto\ssend\smulticast\sUDP\straffic\sbetween\snodes' -ginkgo.flakeAttempts 2 '-ginkgo.skip=recovering from deleting db files while maintain connectivity|Should validate connectivity before and after deleting all the db-pods at once in HA mode|Should be allowed to node local cluster-networked endpoints by nodeport services with externalTrafficPolicy=local|e2e ingress to host-networked pods traffic validation|host to host-networked pods traffic validation' -provider skeleton -kubeconfig /root/ovn.conf --num-nodes=2 --report-dir=/root/ovn-kubernetes/test/_artifacts --report-prefix=control-plane_
I0817 15:26:21.762483 1197731 test_context.go:457] Tolerating taints "node-role.kubernetes.io/control-plane" when considering if nodes are ready
=== RUN TestE2e
I0817 15:26:21.762635 1197731 e2e_suite_test.go:67] Saving reports to /root/ovn-kubernetes/test/_artifacts
Running Suite: E2e Suite
(...)
• [SLOW TEST:12.332 seconds]
Multicast
/root/ovn-kubernetes/test/e2e/multicast.go:25
should be able to send multicast UDP traffic between nodes
/root/ovn-kubernetes/test/e2e/multicast.go:75
------------------------------
SSSSSSSSSSSS
JUnit report was created: /root/ovn-kubernetes/test/_artifacts/junit_control-plane_01.xml
Ran 1 of 60 Specs in 12.333 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Flaked | 0 Pending | 59 Skipped
--- PASS: TestE2e (12.34s)
PASS
ok github.com/ovn-org/ovn-kubernetes/test/e2e 12.371s
+ popd
~/ovn-kubernetes/test