Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2538/e2e-tests/logs/proxy-switch-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster9" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0702 02:53:19.139090 27283 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1782960794323519000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0702 02:53:20.650689 27645 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1782960799323546000&timeoutSeconds=477&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0702 02:53:22.293081 27902 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1782960799323546000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-59848dbdcc-xjngf condition met E0702 02:53:39.710997 29944 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-xjngf&resourceVersion=1782960819351306000&timeoutSeconds=322&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-59848dbdcc-xjngf condition met E0702 02:53:43.445290 30561 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-xjngf&resourceVersion=1782960822621760000&timeoutSeconds=353&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-59848dbdcc-xjngf to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxy-switch-30908 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxy-switch-30908" not found waiting for namespace/proxy-switch-30908 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxy-switch-30908" not found ----------------------------------------------------------------------------------- create namespace proxy-switch-30908 ----------------------------------------------------------------------------------- namespace/proxy-switch-30908 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster9" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/proxy-switch created ----------------------------------------------------------------------------------- waiting pods to be running for proxy-switch ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0702 02:58:29.825515 7117 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1782961108435037000&timeoutSeconds=346&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0702 02:58:42.088226 8943 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1782961118436381000&timeoutSeconds=432&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0702 02:58:55.388119 10964 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1782961133436522000&timeoutSeconds=360&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 02:59:47.253446 18771 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961184130961000&timeoutSeconds=388&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:00:15.579564 23119 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961212924594000&timeoutSeconds=495&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:01:11.300052 31464 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961268452617000&timeoutSeconds=366&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:01:25.212594 1424 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961284129557000&timeoutSeconds=561&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:01:44.301159 3717 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961300705134000&timeoutSeconds=555&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0702 03:02:09.209570 7330 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1782961325705832000&timeoutSeconds=436&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0702 03:02:24.070136 9527 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1782961340706180000&timeoutSeconds=358&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0702 03:02:37.341828 11377 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1782961355707310000&timeoutSeconds=375&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0702 03:02:47.689264 13048 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1782961365707386000&timeoutSeconds=356&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0702 03:03:01.827939 15079 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1782961380707766000&timeoutSeconds=304&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0702 03:03:17.610359 17203 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1782961395708425000&timeoutSeconds=414&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:03:53.194539 22438 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961429936880000&timeoutSeconds=413&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:04:11.979686 25588 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961450830991000&timeoutSeconds=596&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ERROR 1062 (23000) at line 1: Duplicate entry '100500' for key 'myApp.PRIMARY' command terminated with exit code 1 pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:04:39.111416 29261 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961475360733000&timeoutSeconds=328&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:04:59.454369 32497 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961496860338000&timeoutSeconds=353&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:05:17.637600 2744 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961514723795000&timeoutSeconds=515&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:05:30.978169 5143 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961530020006000&timeoutSeconds=520&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- switch from HAProxy to ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for ProxySQL to be ready and HAProxy to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-haproxy to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-haproxy" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-proxysql ----------------------------------------------------------------------------------- pod/proxy-switch-proxysql-0 condition met waiting for pod/proxy-switch-proxysql-0 to become Ready.Ok pod/proxy-switch-proxysql-1 condition met waiting for pod/proxy-switch-proxysql-1 to become Ready.Ok pod/proxy-switch-proxysql-2 condition met waiting for pod/proxy-switch-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify ProxySQL service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-proxysql ClusterIP 34.118.225.51 3306/TCP,33062/TCP,6070/TCP 96s ----------------------------------------------------------------------------------- check connectivity through ProxySQL after switch ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:07:46.161317 24904 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961662984082000&timeoutSeconds=340&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- write new data through ProxySQL ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:08:00.944824 27188 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961678391451000&timeoutSeconds=310&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:08:21.870106 30487 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961700951838000&timeoutSeconds=403&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify new data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:08:35.821607 32347 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961712452008000&timeoutSeconds=539&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:08:54.693904 2700 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961730585374000&timeoutSeconds=360&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:09:15.352911 5776 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961752022468000&timeoutSeconds=565&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify ProxySQL is routing to primary ----------------------------------------------------------------------------------- ProxySQL is routing to primary: proxy-switch-pxc-0 ----------------------------------------------------------------------------------- switch back to HAProxy from ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for HAProxy to be ready and ProxySQL to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-proxysql to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-proxysql" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify ProxySQL pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0702 03:11:13.081716 24155 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1782961870147955000&timeoutSeconds=441&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0702 03:11:22.110268 25587 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1782961880148051000&timeoutSeconds=332&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0702 03:11:33.386502 27165 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1782961890148334000&timeoutSeconds=503&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0702 03:11:42.748749 29004 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1782961900148465000&timeoutSeconds=381&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0702 03:11:53.973164 30404 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1782961910148646000&timeoutSeconds=315&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0702 03:12:06.378711 32259 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1782961925148817000&timeoutSeconds=535&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-haproxy ClusterIP 34.118.237.25 3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP 3m1s ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:12:37.081739 4584 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961954344525000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:12:51.721017 6823 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961970152288000&timeoutSeconds=547&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:13:12.527817 9891 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782961990468363000&timeoutSeconds=527&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:13:23.545008 11843 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782962002532819000&timeoutSeconds=572&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:13:33.789815 13063 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782962011167396000&timeoutSeconds=441&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-xgk5q condition met E0702 03:13:47.389671 15267 reflector.go:227] "Failed to watch" err="Get \"https://34.60.240.116/api/v1/namespaces/proxy-switch-30908/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-xgk5q&resourceVersion=1782962025299477000&timeoutSeconds=415&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-xgk5q to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- clean up ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched perconaxtradbcluster.pxc.percona.com "proxy-switch" deleted from proxy-switch-30908 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "proxy-switch-30908" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------