Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2467/e2e-tests/logs/cross-site-proxysql-8-4.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- Create source cluster ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "pxc" + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc-backup" error: the server doesn't have a resource type "pxc-restore" error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-9d9fbdb5-hjpvd condition met E0516 18:14:20.665296 25926 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-hjpvd&resourceVersion=1778955260306853000&timeoutSeconds=472&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-9d9fbdb5-hjpvd condition met E0516 18:14:31.527258 27342 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-hjpvd&resourceVersion=1778955267688085000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-9d9fbdb5-hjpvd to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-18705 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-18705" not found waiting for namespace/cross-site-proxysql-18705 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "cross-site-proxysql-18705" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-18705 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-18705 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Sat May 16 18:15:23 2026 NAMESPACE: cross-site-proxysql-18705 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.cross-site-proxysql-18705.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace cross-site-proxysql-18705 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace cross-site-proxysql-18705 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace cross-site-proxysql-18705 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace cross-site-proxysql-18705 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-5fd5489bdc-vs894 condition met E0516 18:16:26.225386 6395 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dminio-service-5fd5489bdc-vs894&resourceVersion=1778955384381793000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/minio-service-5fd5489bdc-vs894 to become Ready.Ok make_bucket: operator-testing make_bucket: operator-testing pod "aws-cli" deleted from cross-site-proxysql-18705 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: Internal error occurred: unable to upgrade connection: container aws-cli not found in pod aws-cli_cross-site-proxysql-18705 ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret unchanged secret/aws-s3-secret unchanged secret/do-spaces-secret unchanged secret/gcp-cs-secret unchanged secret/azure-secret unchanged ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-source created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- Error from server (NotFound): pods "cross-site-source-proxysql-0" not found waiting for pod/cross-site-source-proxysql-0 to become Ready....................Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-source-pxc-0 condition met waiting for pod/cross-site-source-pxc-0 to become Ready.Ok pod/cross-site-source-pxc-1 condition met waiting for pod/cross-site-source-pxc-1 to become Ready.Ok pod/cross-site-source-pxc-2 condition met waiting for pod/cross-site-source-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:21:02.140291 27322 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955659554166000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:21:18.533837 29657 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955675721400000&timeoutSeconds=445&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:22:02.297051 2810 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955721527612000&timeoutSeconds=591&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:22:10.008174 3703 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955728127894000&timeoutSeconds=592&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:22:24.507991 5716 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955742066566000&timeoutSeconds=422&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- get main cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:24:02.520271 19688 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955839990818000&timeoutSeconds=414&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch source cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched ----------------------------------------------------------------------------------- patch main cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready ----------------------------------------------------------------------------------- write data to source cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:25:00.310980 28032 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955897859664000&timeoutSeconds=319&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:25:13.935811 30038 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778955911538745000&timeoutSeconds=520&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- take backup of source cluster ----------------------------------------------------------------------------------- [2026-05-16T18:25:24+0000] run pxc-backup/backup-minio-source perconaxtradbclusterbackup.pxc.percona.com/backup-minio-source created waiting for pxc-backup/backup-minio-source to reach Succeeded state................Succeeded ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-replica-3964 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-replica-3964" not found waiting for namespace/cross-site-proxysql-replica-3964 to be deletedError from server (NotFound): namespaces "cross-site-proxysql-replica-3964" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-replica-3964 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-replica-3964 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-9d9fbdb5-rg62p condition met pod/percona-xtradb-cluster-operator-9d9fbdb5-hjpvd condition met E0516 18:27:03.747395 13529 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-hjpvd&resourceVersion=1778956021322535000&timeoutSeconds=535&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-9d9fbdb5-hjpvd to become Ready.Ok secret/cross-site-replica-ssl-internal created ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-replica created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-proxysql-0 condition met waiting for pod/cross-site-replica-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-pxc-0 condition met waiting for pod/cross-site-replica-pxc-0 to become Ready.Ok pod/cross-site-replica-pxc-1 condition met waiting for pod/cross-site-replica-pxc-1 to become Ready.Ok pod/cross-site-replica-pxc-2 condition met waiting for pod/cross-site-replica-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:31:42.219466 18108 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956299716382000&timeoutSeconds=599&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:31:56.113279 19985 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956313588215000&timeoutSeconds=306&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:32:37.717400 25153 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956356165624000&timeoutSeconds=448&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:32:43.923891 25933 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956362068945000&timeoutSeconds=513&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:32:51.731129 27085 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956370983490000&timeoutSeconds=452&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- restore backup from source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready...................................................... ----------------------------------------------------------------------------------- get replica cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:40:41.377448 21459 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956838471447000&timeoutSeconds=355&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:41:03.334575 24770 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956861065418000&timeoutSeconds=367&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch replica cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- patch replica cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- Check replication works between source -> replica ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:42:25.994210 3849 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956944326289000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:42:37.990619 5623 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956955866150000&timeoutSeconds=455&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:42:55.142176 8141 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778956971934333000&timeoutSeconds=589&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check that ProxySQL on replica cluster does not have writer hostgroup (11) ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify cluster stability and no ProxySQL pod restarts over 60 seconds ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- check read access via ProxySQL on replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:44:52.745441 25182 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957090004477000&timeoutSeconds=324&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok [2026-05-16T18:45:02+0000] run pxc-backup/backup-minio-replica perconaxtradbclusterbackup.pxc.percona.com/backup-minio-replica created waiting for pxc-backup/backup-minio-replica to reach Succeeded state..............Succeeded ----------------------------------------------------------------------------------- Switch clusters over ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- rebuild source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready................................................... pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:52:29.129024 26464 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778957545421708000&timeoutSeconds=566&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:52:42.269895 28435 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778957559343079000&timeoutSeconds=502&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- configure old replica as source ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- wait for promoted cluster to be ready ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready... ----------------------------------------------------------------------------------- check that ProxySQL on promoted cluster has writer (11) and reader (10) hostgroups ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- write data via ProxySQL on promoted cluster and verify ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:54:58.898321 15156 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957696224337000&timeoutSeconds=394&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:55:09.621033 16783 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957706991449000&timeoutSeconds=382&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:55:19.569454 18259 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957717645474000&timeoutSeconds=342&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- configure old source as replica ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched perconaxtradbcluster.pxc.percona.com/cross-site-source patched Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. ----------------------------------------------------------------------------------- Write data to replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:55:41.193259 21321 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957738068361000&timeoutSeconds=457&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-2b6jn condition met E0516 18:55:51.945555 23243 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-replica-3964/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-2b6jn&resourceVersion=1778957750910950000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-2b6jn to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- Check replication works between replica -> source ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster7" modified. pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:56:21.153406 27468 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778957778635437000&timeoutSeconds=433&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:56:38.003106 29812 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778957794805226000&timeoutSeconds=334&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-m7g2t condition met E0516 18:56:51.604460 31920 reflector.go:227] "Failed to watch" err="Get \"https://35.253.94.202/api/v1/namespaces/cross-site-proxysql-18705/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-m7g2t&resourceVersion=1778957809563835000&timeoutSeconds=544&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-m7g2t to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- + kubectl patch pxc -n cross-site-proxysql-18705 cross-site-source --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-source patched + kubectl patch pxc -n cross-site-proxysql-replica-3964 cross-site-replica --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com "cross-site-source" deleted from cross-site-proxysql-18705 namespace perconaxtradbcluster.pxc.percona.com "cross-site-replica" deleted from cross-site-proxysql-replica-3964 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-source" deleted from cross-site-proxysql-18705 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-replica" deleted from cross-site-proxysql-replica-3964 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-18705 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-replica-3964 namespace validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "percona-xtradbcluster-webhook" not found ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------