Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2473/e2e-tests/logs/cross-site-proxysql-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- Create source cluster ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "pxc" + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc-backup" error: the server doesn't have a resource type "pxc-restore" error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-55d95dc9d8-wqntl condition met E0516 22:31:11.670677 2824 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-55d95dc9d8-wqntl&resourceVersion=1778970671222269000&timeoutSeconds=579&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-55d95dc9d8-wqntl condition met E0516 22:31:22.730836 4314 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-55d95dc9d8-wqntl&resourceVersion=1778970679436079000&timeoutSeconds=454&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-55d95dc9d8-wqntl to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-8787 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-8787" not found waiting for namespace/cross-site-proxysql-8787 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "cross-site-proxysql-8787" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-8787 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-8787 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "chaos-mesh" chart repository ...Successfully got an update from the "hashicorp" chart repository ...Successfully got an update from the "percona" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Sat May 16 22:32:23 2026 NAMESPACE: cross-site-proxysql-8787 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.cross-site-proxysql-8787.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace cross-site-proxysql-8787 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace cross-site-proxysql-8787 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace cross-site-proxysql-8787 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace cross-site-proxysql-8787 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-5fd5489bdc-lvwv5 condition met E0516 22:33:25.740534 15158 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dminio-service-5fd5489bdc-lvwv5&resourceVersion=1778970804182290000&timeoutSeconds=419&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/minio-service-5fd5489bdc-lvwv5 to become Ready.Ok make_bucket: operator-testing make_bucket: operator-testing pod "aws-cli" deleted from cross-site-proxysql-8787 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: Internal error occurred: unable to upgrade connection: container aws-cli not found in pod aws-cli_cross-site-proxysql-8787 ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret unchanged secret/aws-s3-secret unchanged secret/do-spaces-secret unchanged secret/gcp-cs-secret unchanged secret/azure-secret unchanged ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-source created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-source-proxysql-0 condition met waiting for pod/cross-site-source-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-source-pxc-0 condition met waiting for pod/cross-site-source-pxc-0 to become Ready.Ok pod/cross-site-source-pxc-1 condition met waiting for pod/cross-site-source-pxc-1 to become Ready.Ok pod/cross-site-source-pxc-2 condition met waiting for pod/cross-site-source-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:38:23.152006 4999 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971101064352000&timeoutSeconds=495&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:38:31.683221 6155 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971109891647000&timeoutSeconds=592&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:39:11.672623 11637 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971149381683000&timeoutSeconds=517&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:39:26.295211 13707 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971163297233000&timeoutSeconds=553&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:39:45.002345 16329 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971182478169000&timeoutSeconds=391&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- get main cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:41:15.895314 29357 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971273247529000&timeoutSeconds=409&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch source cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched ----------------------------------------------------------------------------------- patch main cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready ----------------------------------------------------------------------------------- write data to source cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:42:12.645708 4869 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971329958733000&timeoutSeconds=593&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 22:42:28.385647 7203 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778971345954143000&timeoutSeconds=364&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- take backup of source cluster ----------------------------------------------------------------------------------- [2026-05-16T22:42:39+0000] run pxc-backup/backup-minio-source perconaxtradbclusterbackup.pxc.percona.com/backup-minio-source created waiting for pxc-backup/backup-minio-source to reach Succeeded state..................Succeeded ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-replica-7672 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-replica-7672" not found waiting for namespace/cross-site-proxysql-replica-7672 to be deletedError from server (NotFound): namespaces "cross-site-proxysql-replica-7672" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-replica-7672 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-replica-7672 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created error: timed out waiting for the condition on pods/percona-xtradb-cluster-operator-55d95dc9d8-9r8fw pod/percona-xtradb-cluster-operator-55d95dc9d8-wqntl condition met E0516 22:44:30.847979 24732 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-55d95dc9d8-wqntl&resourceVersion=1778971469210511003&timeoutSeconds=320&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-55d95dc9d8-wqntl to become Ready.Ok secret/cross-site-replica-ssl-internal created ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-replica created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-proxysql-0 condition met waiting for pod/cross-site-replica-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-pxc-0 condition met waiting for pod/cross-site-replica-pxc-0 to become Ready.Ok pod/cross-site-replica-pxc-1 condition met waiting for pod/cross-site-replica-pxc-1 to become Ready.Ok pod/cross-site-replica-pxc-2 condition met waiting for pod/cross-site-replica-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:49:22.858081 31506 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778971760610533000&timeoutSeconds=540&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:49:33.931145 652 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778971773749199003&timeoutSeconds=461&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:50:16.224694 5995 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778971814311531000&timeoutSeconds=579&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:50:22.448543 6985 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778971821666119000&timeoutSeconds=479&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:50:30.546863 8007 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778971828675661000&timeoutSeconds=333&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- restore backup from source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready................................................. ----------------------------------------------------------------------------------- get replica cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:57:42.361033 29543 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972260163120000&timeoutSeconds=455&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:57:58.673970 31949 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972276453990000&timeoutSeconds=351&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch replica cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- patch replica cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- Check replication works between source -> replica ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:59:23.030127 11555 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972361415460000&timeoutSeconds=526&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:59:34.520068 13239 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972372143272000&timeoutSeconds=459&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 22:59:45.489940 14836 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972383590924000&timeoutSeconds=389&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check that ProxySQL on replica cluster does not have writer hostgroup (11) ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify cluster stability and no ProxySQL pod restarts over 60 seconds ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- check read access via ProxySQL on replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:01:34.720816 30805 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778972493857260000&timeoutSeconds=478&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok [2026-05-16T23:01:41+0000] run pxc-backup/backup-minio-replica perconaxtradbclusterbackup.pxc.percona.com/backup-minio-replica created waiting for pxc-backup/backup-minio-replica to reach Succeeded state...........Succeeded ----------------------------------------------------------------------------------- Switch clusters over ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- rebuild source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready..................................................... pod/pxc-client-67fc4995bb-2bqnz condition met E0516 23:08:54.143538 27775 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778972932433419000&timeoutSeconds=452&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 23:09:03.322576 28984 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778972941443722000&timeoutSeconds=347&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- configure old replica as source ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- wait for promoted cluster to be ready ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready..... ----------------------------------------------------------------------------------- check that ProxySQL on promoted cluster has writer (11) and reader (10) hostgroups ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- write data via ProxySQL on promoted cluster and verify ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:11:22.792970 14893 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778973080440506000&timeoutSeconds=389&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:11:35.604491 16685 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778973092726179000&timeoutSeconds=516&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:11:49.650899 18643 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778973107369838000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- configure old source as replica ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched perconaxtradbcluster.pxc.percona.com/cross-site-source patched Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. ----------------------------------------------------------------------------------- Write data to replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:12:03.163042 20575 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778973121263979000&timeoutSeconds=345&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-r99xv condition met E0516 23:12:17.894539 22570 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-replica-7672/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-r99xv&resourceVersion=1778973136237759020&timeoutSeconds=524&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-r99xv to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- Check replication works between replica -> source ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2473-6d392bea-4-cluster6" modified. pod/pxc-client-67fc4995bb-2bqnz condition met E0516 23:12:46.902720 26879 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778973166239199003&timeoutSeconds=319&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 23:13:00.246421 28761 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778973177350331000&timeoutSeconds=322&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-2bqnz condition met E0516 23:13:11.614837 30493 reflector.go:227] "Failed to watch" err="Get \"https://35.225.104.245/api/v1/namespaces/cross-site-proxysql-8787/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-2bqnz&resourceVersion=1778973189596888000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-2bqnz to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- + kubectl patch pxc -n cross-site-proxysql-8787 cross-site-source --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-source patched + kubectl patch pxc -n cross-site-proxysql-replica-7672 cross-site-replica --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com "cross-site-source" deleted from cross-site-proxysql-8787 namespace perconaxtradbcluster.pxc.percona.com "cross-site-replica" deleted from cross-site-proxysql-replica-7672 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-source" deleted from cross-site-proxysql-8787 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-replica" deleted from cross-site-proxysql-replica-7672 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-8787 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-replica-7672 namespace validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "percona-xtradbcluster-webhook" not found ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------