Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2467/e2e-tests/logs/cross-site-proxysql-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- Create source cluster ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "pxc" + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc-backup" error: the server doesn't have a resource type "pxc-restore" error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-9d9fbdb5-bfd26 condition met E0516 18:14:02.421973 23432 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-bfd26&resourceVersion=1778955242056701000&timeoutSeconds=418&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-9d9fbdb5-bfd26 condition met E0516 18:14:10.623433 24568 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-bfd26&resourceVersion=1778955247381547000&timeoutSeconds=542&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-9d9fbdb5-bfd26 to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-14234 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-14234" not found waiting for namespace/cross-site-proxysql-14234 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "cross-site-proxysql-14234" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-14234 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-14234 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Sat May 16 18:15:09 2026 NAMESPACE: cross-site-proxysql-14234 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.cross-site-proxysql-14234.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace cross-site-proxysql-14234 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace cross-site-proxysql-14234 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace cross-site-proxysql-14234 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace cross-site-proxysql-14234 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-5fd5489bdc-v6dhh condition met E0516 18:16:12.150689 5583 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dminio-service-5fd5489bdc-v6dhh&resourceVersion=1778955370698635000&timeoutSeconds=590&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/minio-service-5fd5489bdc-v6dhh to become Ready.Ok pod "aws-cli" deleted from cross-site-proxysql-14234 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret unchanged secret/aws-s3-secret unchanged secret/do-spaces-secret unchanged secret/gcp-cs-secret unchanged secret/azure-secret unchanged ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-source created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-source-proxysql-0 condition met waiting for pod/cross-site-source-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-source-pxc-0 condition met waiting for pod/cross-site-source-pxc-0 to become Ready.Ok pod/cross-site-source-pxc-1 condition met waiting for pod/cross-site-source-pxc-1 to become Ready.Ok pod/cross-site-source-pxc-2 condition met waiting for pod/cross-site-source-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:21:13.843310 28898 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955670865706000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:21:31.015557 31547 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955688555426000&timeoutSeconds=385&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:22:11.099164 3877 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955729141923000&timeoutSeconds=350&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:22:26.158403 5942 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955743672242000&timeoutSeconds=410&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:22:37.553816 7863 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955756583188000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- get main cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:24:12.542488 21046 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955849398786000&timeoutSeconds=563&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch source cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched ----------------------------------------------------------------------------------- patch main cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready ----------------------------------------------------------------------------------- write data to source cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:25:09.184197 29400 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955906826770000&timeoutSeconds=330&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:25:22.757619 31350 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778955920317687000&timeoutSeconds=516&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- take backup of source cluster ----------------------------------------------------------------------------------- [2026-05-16T18:25:34+0000] run pxc-backup/backup-minio-source perconaxtradbclusterbackup.pxc.percona.com/backup-minio-source created waiting for pxc-backup/backup-minio-source to reach Succeeded state...............Succeeded ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-replica-32336 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-replica-32336" not found waiting for namespace/cross-site-proxysql-replica-32336 to be deletedError from server (NotFound): namespaces "cross-site-proxysql-replica-32336" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-replica-32336 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-replica-32336 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created error: timed out waiting for the condition on pods/percona-xtradb-cluster-operator-9d9fbdb5-ctwkl pod/percona-xtradb-cluster-operator-9d9fbdb5-bfd26 condition met E0516 18:27:27.878317 17035 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-bfd26&resourceVersion=1778956045184886000&timeoutSeconds=390&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-9d9fbdb5-bfd26 to become Ready.Ok secret/cross-site-replica-ssl-internal created ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-replica created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-proxysql-0 condition met waiting for pod/cross-site-replica-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/cross-site-replica-pxc-0 condition met waiting for pod/cross-site-replica-pxc-0 to become Ready.Ok pod/cross-site-replica-pxc-1 condition met waiting for pod/cross-site-replica-pxc-1 to become Ready.Ok pod/cross-site-replica-pxc-2 condition met waiting for pod/cross-site-replica-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:32:18.855512 22930 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956337036752000&timeoutSeconds=351&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:32:25.076785 23721 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956343239894000&timeoutSeconds=327&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:33:04.474981 28457 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956382760142000&timeoutSeconds=366&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:33:10.518353 29367 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956389569690000&timeoutSeconds=319&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:33:18.236914 30183 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956396529399000&timeoutSeconds=552&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- restore backup from source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready.................................................... ----------------------------------------------------------------------------------- get replica cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:40:57.003195 23734 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956854244093000&timeoutSeconds=348&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:41:14.713625 26487 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956872499231000&timeoutSeconds=421&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch replica cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- patch replica cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- Check replication works between source -> replica ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:42:39.165620 6014 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956958149213000&timeoutSeconds=431&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:42:56.310855 8253 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956972925891000&timeoutSeconds=581&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:43:13.123600 10769 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778956991454911009&timeoutSeconds=503&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check that ProxySQL on replica cluster does not have writer hostgroup (11) ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify cluster stability and no ProxySQL pod restarts over 60 seconds ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- check read access via ProxySQL on replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:45:01.540550 26466 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957098258565000&timeoutSeconds=443&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok [2026-05-16T18:45:09+0000] run pxc-backup/backup-minio-replica perconaxtradbclusterbackup.pxc.percona.com/backup-minio-replica created waiting for pxc-backup/backup-minio-replica to reach Succeeded state...............Succeeded ----------------------------------------------------------------------------------- Switch clusters over ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- rebuild source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready................................................. pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:52:12.525456 24149 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778957528944909000&timeoutSeconds=314&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:52:30.813291 26870 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778957548035092000&timeoutSeconds=514&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- configure old replica as source ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- wait for promoted cluster to be ready ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready..... ----------------------------------------------------------------------------------- check that ProxySQL on promoted cluster has writer (11) and reader (10) hostgroups ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- write data via ProxySQL on promoted cluster and verify ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:55:02.293958 15662 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957699795707000&timeoutSeconds=384&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:55:10.884322 17102 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957709937499000&timeoutSeconds=347&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:55:20.605065 18372 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957718479979000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- configure old source as replica ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched perconaxtradbcluster.pxc.percona.com/cross-site-source patched Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. ----------------------------------------------------------------------------------- Write data to replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:55:41.354894 21384 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957738312656000&timeoutSeconds=546&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-ks9vh condition met E0516 18:55:57.875349 23868 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-replica-32336/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-ks9vh&resourceVersion=1778957756990959018&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-ks9vh to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- Check replication works between replica -> source ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster6" modified. pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:56:27.859980 28478 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778957785203379000&timeoutSeconds=506&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:56:46.011388 31035 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778957803239184000&timeoutSeconds=537&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-8wfv2 condition met E0516 18:56:58.154446 439 reflector.go:227] "Failed to watch" err="Get \"https://34.56.22.232/api/v1/namespaces/cross-site-proxysql-14234/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-8wfv2&resourceVersion=1778957816006640000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-8wfv2 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- + kubectl patch pxc -n cross-site-proxysql-14234 cross-site-source --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-source patched + kubectl patch pxc -n cross-site-proxysql-replica-32336 cross-site-replica --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com "cross-site-source" deleted from cross-site-proxysql-14234 namespace perconaxtradbcluster.pxc.percona.com "cross-site-replica" deleted from cross-site-proxysql-replica-32336 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-source" deleted from cross-site-proxysql-14234 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-replica" deleted from cross-site-proxysql-replica-32336 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-14234 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-replica-32336 namespace validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "percona-xtradbcluster-webhook" not found ----------------------------------------------------------------------------------- test passed ----------------------------------------------------------------------------------- namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): namespaces "pxc-operator" not found