Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2521/e2e-tests/logs/proxy-switch-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster12" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0611 02:53:58.296465 17156 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1781146435238376000&timeoutSeconds=407&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0611 02:54:02.294291 17790 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1781146440238387000&timeoutSeconds=443&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0611 02:54:06.344299 18382 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1781146445238403000&timeoutSeconds=573&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-774dd4b69-k9dbg condition met E0611 02:54:33.049309 22395 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-k9dbg&resourceVersion=1781146472690766000&timeoutSeconds=523&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-774dd4b69-k9dbg condition met E0611 02:54:40.414448 23871 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-k9dbg&resourceVersion=1781146479166191000&timeoutSeconds=325&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-774dd4b69-k9dbg to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxy-switch-11120 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxy-switch-11120" not found waiting for namespace/proxy-switch-11120 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxy-switch-11120" not found ----------------------------------------------------------------------------------- create namespace proxy-switch-11120 ----------------------------------------------------------------------------------- namespace/proxy-switch-11120 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster12" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/proxy-switch created ----------------------------------------------------------------------------------- waiting pods to be running for proxy-switch ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0611 02:59:21.309530 1210 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1781146761090138000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0611 02:59:34.306756 3171 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1781146771090179000&timeoutSeconds=430&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0611 02:59:47.732213 5256 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1781146786090224000&timeoutSeconds=416&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:00:27.712309 11688 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781146825440665000&timeoutSeconds=505&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:00:50.278001 15114 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781146847445962000&timeoutSeconds=426&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:01:46.382656 23690 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781146903151154000&timeoutSeconds=580&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:02:02.902452 26092 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781146918888222000&timeoutSeconds=566&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:02:22.333676 29118 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781146938639218000&timeoutSeconds=311&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0611 03:02:39.565734 31877 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1781146958639751000&timeoutSeconds=519&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0611 03:02:48.718147 838 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1781146963639850000&timeoutSeconds=531&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0611 03:02:56.639093 2017 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1781146973639908000&timeoutSeconds=543&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0611 03:03:03.849379 3079 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1781146983639923000&timeoutSeconds=491&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0611 03:03:13.676504 4432 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1781146988640000000&timeoutSeconds=477&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0611 03:03:28.245139 6797 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1781147003640263000&timeoutSeconds=369&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:04:04.094183 12361 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147039676437000&timeoutSeconds=348&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:04:28.835292 16105 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147064524166000&timeoutSeconds=354&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ERROR 1062 (23000) at line 1: Duplicate entry '100500' for key 'myApp.PRIMARY' command terminated with exit code 1 pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:04:56.912044 20372 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147093756956000&timeoutSeconds=443&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:05:14.374409 23076 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147111431213000&timeoutSeconds=365&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:05:25.916655 24894 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147123598911000&timeoutSeconds=336&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:05:34.134563 26248 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147132643484000&timeoutSeconds=315&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- switch from HAProxy to ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for ProxySQL to be ready and HAProxy to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-haproxy to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-haproxy" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-proxysql ----------------------------------------------------------------------------------- pod/proxy-switch-proxysql-0 condition met waiting for pod/proxy-switch-proxysql-0 to become Ready.Ok pod/proxy-switch-proxysql-1 condition met waiting for pod/proxy-switch-proxysql-1 to become Ready.Ok pod/proxy-switch-proxysql-2 condition met waiting for pod/proxy-switch-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify ProxySQL service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-proxysql ClusterIP 34.118.237.49 3306/TCP,33062/TCP,6070/TCP 105s ----------------------------------------------------------------------------------- check connectivity through ProxySQL after switch ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:07:51.303698 14751 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147269732058000&timeoutSeconds=484&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- write new data through ProxySQL ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:08:03.288452 16636 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147282327026000&timeoutSeconds=470&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:08:26.659777 19921 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147303985694000&timeoutSeconds=415&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify new data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:08:45.269861 22657 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147322540298000&timeoutSeconds=524&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:09:04.481544 25871 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147343400051000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:09:29.264994 28928 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147365555121000&timeoutSeconds=393&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify ProxySQL is routing to primary ----------------------------------------------------------------------------------- ProxySQL is routing to primary: proxy-switch-pxc-0 ----------------------------------------------------------------------------------- switch back to HAProxy from ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for HAProxy to be ready and ProxySQL to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-proxysql to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-proxysql" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify ProxySQL pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0611 03:11:27.518931 14508 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1781147484547163000&timeoutSeconds=365&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0611 03:11:34.409744 15611 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1781147489547761000&timeoutSeconds=388&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0611 03:11:42.637439 16871 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1781147499547821000&timeoutSeconds=309&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0611 03:11:52.028470 18163 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1781147509547968000&timeoutSeconds=344&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0611 03:12:04.034117 19940 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1781147519548046000&timeoutSeconds=501&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0611 03:12:11.241340 21192 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1781147529548128000&timeoutSeconds=469&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-haproxy ClusterIP 34.118.236.14 3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP 2m43s ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:12:40.727133 25690 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147557402610000&timeoutSeconds=496&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:12:55.093558 28041 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147572980432000&timeoutSeconds=407&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:13:17.574871 31430 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147594929576000&timeoutSeconds=465&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:13:36.427021 1724 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147612750705000&timeoutSeconds=498&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:13:54.889752 4771 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147632146372000&timeoutSeconds=481&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-jx9r4 condition met E0611 03:14:12.739004 7522 reflector.go:227] "Failed to watch" err="Get \"https://34.56.75.63/api/v1/namespaces/proxy-switch-11120/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-jx9r4&resourceVersion=1781147649530563000&timeoutSeconds=305&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-jx9r4 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- clean up ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched perconaxtradbcluster.pxc.percona.com "proxy-switch" deleted from proxy-switch-11120 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "proxy-switch-11120" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------