Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2473/e2e-tests/logs/proxy-switch-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster3" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-55d95dc9d8-48brd condition met E0517 00:17:20.035229 15720 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-55d95dc9d8-48brd&resourceVersion=1778977039673117000&timeoutSeconds=411&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-55d95dc9d8-48brd condition met E0517 00:17:28.205393 16860 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-55d95dc9d8-48brd&resourceVersion=1778977045771355000&timeoutSeconds=437&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-55d95dc9d8-48brd to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxy-switch-22923 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxy-switch-22923" not found waiting for namespace/proxy-switch-22923 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxy-switch-22923" not found ----------------------------------------------------------------------------------- create namespace proxy-switch-22923 ----------------------------------------------------------------------------------- namespace/proxy-switch-22923 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster3" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/proxy-switch created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0517 00:19:20.932058 31644 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778977159942159011&timeoutSeconds=468&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:22:09.573976 15063 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977328793277000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:22:15.341012 15705 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977334571417000&timeoutSeconds=405&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:22:54.978693 19847 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977372620100000&timeoutSeconds=369&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:23:08.436775 21840 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977385994803000&timeoutSeconds=547&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:23:18.805585 23442 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977397363447000&timeoutSeconds=415&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0517 00:23:27.174999 24432 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778977402363544000&timeoutSeconds=444&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0517 00:23:31.157247 24940 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1778977407363565000&timeoutSeconds=579&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0517 00:23:36.554428 25398 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1778977412363619000&timeoutSeconds=487&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0517 00:23:43.140429 26241 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1778977422364587000&timeoutSeconds=385&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0517 00:23:51.747336 27314 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1778977427364703000&timeoutSeconds=543&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0517 00:24:02.321637 28935 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1778977437364754000&timeoutSeconds=337&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:24:32.854899 1047 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977470346181000&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:24:42.627042 2546 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977480933701000&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ERROR 1062 (23000) at line 1: Duplicate entry '100500' for key 'myApp.PRIMARY' command terminated with exit code 1 pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:25:02.457016 5054 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977500681181000&timeoutSeconds=476&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:25:10.607152 6107 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977508647783000&timeoutSeconds=338&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:25:22.820526 7826 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977520253463000&timeoutSeconds=481&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:25:34.740544 9555 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977532385655000&timeoutSeconds=459&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- switch from HAProxy to ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for ProxySQL to be ready and HAProxy to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-haproxy to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-haproxy" not found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-proxysql-0 condition met waiting for pod/proxy-switch-proxysql-0 to become Ready.Ok pod/proxy-switch-proxysql-1 condition met waiting for pod/proxy-switch-proxysql-1 to become Ready.Ok pod/proxy-switch-proxysql-2 condition met waiting for pod/proxy-switch-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify ProxySQL service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-proxysql ClusterIP 34.118.226.22 3306/TCP,33062/TCP,6070/TCP 84s ----------------------------------------------------------------------------------- check connectivity through ProxySQL after switch ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:27:31.022536 25647 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977649219919000&timeoutSeconds=411&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- write new data through ProxySQL ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:27:41.215062 27085 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977659196362000&timeoutSeconds=319&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:28:04.821477 30544 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977682703066000&timeoutSeconds=375&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify new data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:28:16.525593 32127 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977693780547000&timeoutSeconds=552&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:28:26.533262 1123 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977704491951000&timeoutSeconds=457&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:28:36.925966 2592 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977714655144000&timeoutSeconds=478&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify ProxySQL is routing to primary ----------------------------------------------------------------------------------- ProxySQL is routing to primary: proxy-switch-pxc-0 ----------------------------------------------------------------------------------- switch back to HAProxy from ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for HAProxy to be ready and ProxySQL to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-proxysql to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-proxysql" not found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify ProxySQL pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0517 00:30:25.798318 17432 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778977823944576000&timeoutSeconds=439&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0517 00:30:32.292442 18308 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1778977828944643000&timeoutSeconds=398&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0517 00:30:39.598422 19347 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1778977838944778000&timeoutSeconds=305&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0517 00:30:47.293184 20380 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1778977843944796000&timeoutSeconds=438&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0517 00:30:55.156998 21510 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1778977853945194000&timeoutSeconds=332&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0517 00:31:02.838100 22580 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1778977858945340000&timeoutSeconds=399&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-haproxy ClusterIP 34.118.235.195 3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP 2m32s ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:31:24.678059 25612 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977882760500000&timeoutSeconds=411&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:31:35.693838 27119 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977893294748000&timeoutSeconds=324&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:31:58.189206 30128 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977915343176000&timeoutSeconds=360&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:32:11.657533 32006 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977929318831000&timeoutSeconds=549&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:32:24.636508 1433 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977942311523000&timeoutSeconds=361&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-hlhzm condition met E0517 00:32:36.985194 3200 reflector.go:227] "Failed to watch" err="Get \"https://34.56.49.150/api/v1/namespaces/proxy-switch-22923/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-hlhzm&resourceVersion=1778977954600384000&timeoutSeconds=485&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-hlhzm to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- clean up ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- + kubectl patch pxc -n proxy-switch-22923 proxy-switch --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/proxy-switch patched perconaxtradbcluster.pxc.percona.com "proxy-switch" deleted from proxy-switch-22923 namespace No resources found No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------