Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2538/e2e-tests/logs/upgrade-haproxy-5-7.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-tokenrequest created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-tokenrequest created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager-cainjector created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. deployment "cert-manager" successfully rolled out deployment "cert-manager-cainjector" successfully rolled out deployment "cert-manager-webhook" successfully rolled out [2026-07-06T11:34:51+0000] waiting for cert-manager webhook caBundle to be injected [2026-07-06T11:34:56+0000] probing cert-manager webhook error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace "cert-manager" deleted namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-2-cluster13" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-774b5698c6-qhggn condition met E0706 11:36:41.169476 7208 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774b5698c6-qhggn&resourceVersion=1783337799142127000&timeoutSeconds=352&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-774b5698c6-qhggn to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces upgrade-haproxy-21598 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "upgrade-haproxy-21598" not found waiting for namespace/upgrade-haproxy-21598 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "upgrade-haproxy-21598" not found ----------------------------------------------------------------------------------- create namespace upgrade-haproxy-21598 ----------------------------------------------------------------------------------- namespace/upgrade-haproxy-21598 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-2-cluster13" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Mon Jul 6 11:37:40 2026 NAMESPACE: upgrade-haproxy-21598 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.upgrade-haproxy-21598.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace upgrade-haproxy-21598 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace upgrade-haproxy-21598 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace upgrade-haproxy-21598 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace upgrade-haproxy-21598 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-5fd5489bdc-5kbzj condition met E0706 11:38:42.964946 21673 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dminio-service-5fd5489bdc-5kbzj&resourceVersion=1783337922179201000&timeoutSeconds=344&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/minio-service-5fd5489bdc-5kbzj to become Ready.Ok make_bucket: operator-testing make_bucket: operator-testing pod "aws-cli" deleted from upgrade-haproxy-21598 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: Internal error occurred: unable to upgrade connection: container aws-cli not found in pod aws-cli_upgrade-haproxy-21598 ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/upgrade-haproxy created ----------------------------------------------------------------------------------- waiting pods to be running for upgrade-haproxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for upgrade-haproxy-pxc ----------------------------------------------------------------------------------- pod/upgrade-haproxy-pxc-0 condition met waiting for pod/upgrade-haproxy-pxc-0 to become Ready.Ok pod/upgrade-haproxy-pxc-1 condition met waiting for pod/upgrade-haproxy-pxc-1 to become Ready.Ok pod/upgrade-haproxy-pxc-2 condition met waiting for pod/upgrade-haproxy-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for upgrade-haproxy-haproxy ----------------------------------------------------------------------------------- pod/upgrade-haproxy-haproxy-0 condition met E0706 11:43:45.703229 26574 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-haproxy-0&resourceVersion=1783338221535018000&timeoutSeconds=347&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-haproxy-0 to become Ready.Ok pod/upgrade-haproxy-haproxy-1 condition met E0706 11:43:53.811048 27945 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-haproxy-1&resourceVersion=1783338231535049000&timeoutSeconds=492&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-haproxy-1 to become Ready.Ok pod/upgrade-haproxy-haproxy-2 condition met E0706 11:44:01.007062 29100 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-haproxy-2&resourceVersion=1783338236535089000&timeoutSeconds=593&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/upgrade-haproxy to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:45:06.888637 7315 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338305813443000&timeoutSeconds=334&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:45:29.047861 10936 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338328057226000&timeoutSeconds=306&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:46:26.839974 20143 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338383330283000&timeoutSeconds=359&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:46:43.711802 22793 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338399748711000&timeoutSeconds=577&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:47:00.952495 25452 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338416934939000&timeoutSeconds=430&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok [2026-07-06T11:47:28+0000] run pxc-backup/on-demand-backup-minio perconaxtradbclusterbackup.pxc.percona.com/on-demand-backup-minio created waiting for pxc-backup/on-demand-backup-minio to reach Succeeded state...........Succeeded ----------------------------------------------------------------------------------- upgrade operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator patched deployment "percona-xtradb-cluster-operator" successfully rolled out ----------------------------------------------------------------------------------- wait for operator upgrade ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-2-cluster13" modified. pod/percona-xtradb-cluster-operator-7d79d5dcbd-shlfp condition met E0706 11:49:11.645193 13140 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-7d79d5dcbd-shlfp&resourceVersion=1783338547030559000&timeoutSeconds=538&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-7d79d5dcbd-shlfp to become Ready.Ok Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-2-cluster13" modified. ----------------------------------------------------------------------------------- check images and generation after operator upgrade ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/upgrade-haproxy to be ready ----------------------------------------------------------------------------------- Waiting for 3 running pods for upgrade-haproxy-pxc ----------------------------------------------------------------------------------- pod/upgrade-haproxy-pxc-0 condition met E0706 11:49:58.207450 20519 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-0&resourceVersion=1783338597031183000&timeoutSeconds=438&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-0 to become Ready.Ok pod/upgrade-haproxy-pxc-1 condition met E0706 11:50:15.910986 23128 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-1&resourceVersion=1783338612031558000&timeoutSeconds=543&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-1 to become Ready.Ok pod/upgrade-haproxy-pxc-2 condition met E0706 11:50:30.334646 25568 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-2&resourceVersion=1783338627031695000&timeoutSeconds=547&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-2 to become Ready.Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:50:48.034824 28269 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338643854999000&timeoutSeconds=418&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:51:14.347272 32395 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338670511485000&timeoutSeconds=436&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:51:39.403058 3900 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783338695774559000&timeoutSeconds=568&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch pxc images and upgrade ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/upgrade-haproxy patched ----------------------------------------------------------------------------------- check images and generation after full upgrade ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/upgrade-haproxy to be ready.......... ----------------------------------------------------------------------------------- Waiting for 3 running pods for upgrade-haproxy-pxc ----------------------------------------------------------------------------------- pod/upgrade-haproxy-pxc-0 condition met E0706 11:56:29.533734 16714 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-0&resourceVersion=1783338986010232000&timeoutSeconds=506&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-0 to become Ready.Ok pod/upgrade-haproxy-pxc-1 condition met E0706 11:56:43.327773 18756 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-1&resourceVersion=1783339001010315000&timeoutSeconds=567&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-1 to become Ready.Ok pod/upgrade-haproxy-pxc-2 condition met E0706 11:56:54.970098 20614 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-2&resourceVersion=1783339011010390000&timeoutSeconds=550&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-2 to become Ready.Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:57:12.278729 23414 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339028431563000&timeoutSeconds=384&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:57:28.783303 25967 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339044373473000&timeoutSeconds=462&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:57:50.168888 29330 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339066441405000&timeoutSeconds=506&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare statefulset/upgrade-haproxy-pxc- ----------------------------------------------------------------------------------- [2026-07-06T11:58:31+0000] compare_kubectl: statefulset/upgrade-haproxy-pxc OK ----------------------------------------------------------------------------------- compare statefulset/upgrade-haproxy-haproxy- ----------------------------------------------------------------------------------- [2026-07-06T11:58:35+0000] compare_kubectl: statefulset/upgrade-haproxy-haproxy OK ----------------------------------------------------------------------------------- check data exists after upgrade and drop database ----------------------------------------------------------------------------------- pod/pxc-client-6c975684f9-xxr5b condition met E0706 11:59:43.047703 14710 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339178846411000&timeoutSeconds=326&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 12:00:03.976850 17909 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339199798819000&timeoutSeconds=457&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 12:00:22.044274 20753 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339218572019000&timeoutSeconds=332&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-xxr5b condition met E0706 12:00:38.917419 23500 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-xxr5b&resourceVersion=1783339235607253000&timeoutSeconds=411&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-xxr5b to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- run restore ----------------------------------------------------------------------------------- [2026-07-06T12:00:56+0000] run pxc-restore/on-demand-backup-minio perconaxtradbclusterrestore.pxc.percona.com/on-demand-backup-minio created waiting for pxc-restore/on-demand-backup-minio to reach Succeeded state 2026-07-06T12:01:04 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:09 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:14 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:19 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:23 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:29 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:34 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:39 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:43 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:47 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:51 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:55 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:01:58 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:01 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:06 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:10 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:15 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:18 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:22 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:25 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:28 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:31 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:34 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:38 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:42 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:45 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:48 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:53 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:02:56 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:00 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:03 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:07 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:10 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:14 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:17 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:20 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:22 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:25 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:28 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:32 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:35 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:39 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:43 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:46 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:49 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:52 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:55 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:03:57 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:04:00 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:04:03 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:04:06 pxc-restore/on-demand-backup-minio state: Stopping Cluster 2026-07-06T12:04:09 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:12 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:14 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:17 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:20 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:23 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:26 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:29 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:31 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:34 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:37 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:39 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:42 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:45 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:48 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:50 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:53 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:55 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:04:58 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:01 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:04 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:06 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:09 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:12 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:15 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:18 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:20 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:23 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:26 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:29 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:31 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:34 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:37 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:40 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:42 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:44 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:47 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:49 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:51 pxc-restore/on-demand-backup-minio state: Restoring 2026-07-06T12:05:54 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:05:57 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:05:59 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:02 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:05 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:08 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:11 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:14 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:16 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:19 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:22 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:25 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:27 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:30 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:32 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:35 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:38 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:40 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:43 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:45 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:47 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:50 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:52 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:54 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:56 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:06:59 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:01 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:03 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:05 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:08 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:10 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:12 pxc-restore/on-demand-backup-minio state: Preparing Cluster 2026-07-06T12:07:14 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:17 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:19 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:22 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:24 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:27 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:29 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:31 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:33 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:36 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:38 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:41 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:43 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:45 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:47 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:50 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:52 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:54 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:56 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:07:59 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:01 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:04 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:06 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:08 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:10 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:13 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:15 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:17 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:20 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:23 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:25 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:28 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:30 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:32 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:34 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:36 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:39 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:42 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:44 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:46 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:49 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:51 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:54 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:56 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:08:59 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:01 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:03 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:05 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:08 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:10 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:12 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:14 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:16 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:19 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:21 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:23 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:25 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:28 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:31 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:33 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:35 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:38 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:41 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:44 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:48 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:51 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:54 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:09:57 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:01 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:05 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:08 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:11 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:15 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:17 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:20 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:23 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:25 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:28 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:31 pxc-restore/on-demand-backup-minio state: Starting Cluster 2026-07-06T12:10:34 pxc-restore/on-demand-backup-minio state: Succeeded + LIB_PATH=/opt/percona/backup/lib/pxc + . /opt/percona/backup/lib/pxc/check-version.sh + . /opt/percona/backup/lib/pxc/vault.sh ++ set -o errexit ++ keyring_vault=/etc/mysql/vault-keyring-secret/keyring_vault.conf + . /opt/percona/backup/lib/pxc/aws.sh ++ set -o errexit ++ export AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile ++ AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile ++ export AWS_REGION=us-east-1 ++ AWS_REGION=us-east-1 ++ export AWS_ENDPOINT_URL=http://minio-service.upgrade-haproxy-21598:9000/ ++ AWS_ENDPOINT_URL=http://minio-service.upgrade-haproxy-21598:9000/ ++ '[' -n true ']' ++ [[ true == \f\a\l\s\e ]] ++ caBundleDir=/etc/s3/certs ++ caBundleFile=/etc/s3/certs/ca.crt ++ '[' -f /etc/s3/certs/ca.crt ']' + XBCLOUD_ARGS='--curl-retriable-errors=7 ' + '[' -n true ']' + [[ true == \f\a\l\s\e ]] + '[' -n operator-testing/upgrade-haproxy-2026-07-06-11:47:32-full ']' + set -x + aws s3 ls operator-testing/upgrade-haproxy-2026-07-06-11:47:32-full Could not connect to the endpoint URL: "http://minio-service.upgrade-haproxy-21598:9000/operator-testing?list-type=2&prefix=upgrade-haproxy-2026-07-06-11%3A47%3A32-full&delimiter=%2F&encoding-type=url" Found 3 pods, using pod/restore-job-on-demand-backup-minio-upgrade-haproxy-tzs56 Defaulted container "xtrabackup" out of: xtrabackup, backup-init (init) ----------------------------------------------------------------------------------- Waiting for 1 running pods for upgrade-haproxy-haproxy ----------------------------------------------------------------------------------- pod/upgrade-haproxy-haproxy-0 condition met E0706 12:10:39.845611 17988 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-haproxy-0&resourceVersion=1783339837707715000&timeoutSeconds=596&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-haproxy-0 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for upgrade-haproxy-pxc ----------------------------------------------------------------------------------- pod/upgrade-haproxy-pxc-0 condition met E0706 12:10:46.231951 19010 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-0&resourceVersion=1783339842707755000&timeoutSeconds=578&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-0 to become Ready.Ok pod/upgrade-haproxy-pxc-1 condition met E0706 12:10:55.163021 20361 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-1&resourceVersion=1783339852707803000&timeoutSeconds=393&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-1 to become Ready.Ok pod/upgrade-haproxy-pxc-2 condition met E0706 12:11:05.499013 21943 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dupgrade-haproxy-pxc-2&resourceVersion=1783339862707995000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/upgrade-haproxy-pxc-2 to become Ready.Ok [2026-07-06T12:11:51+0000] check data after pxc-restore/on-demand-backup-minio pod/pxc-client-6c975684f9-j6w9p condition met E0706 12:12:01.542043 30355 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-j6w9p&resourceVersion=1783339919016264000&timeoutSeconds=339&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-j6w9p to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-j6w9p condition met E0706 12:12:14.660531 32409 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-j6w9p&resourceVersion=1783339932421780000&timeoutSeconds=388&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-j6w9p to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-6c975684f9-j6w9p condition met E0706 12:12:26.540538 1789 reflector.go:227] "Failed to watch" err="Get \"https://34.16.9.70/api/v1/namespaces/upgrade-haproxy-21598/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-6c975684f9-j6w9p&resourceVersion=1783339944331255000&timeoutSeconds=338&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-6c975684f9-j6w9p to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- cleanup ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbclusterbackup.pxc.percona.com "on-demand-backup-minio" deleted from upgrade-haproxy-21598 namespace perconaxtradbcluster.pxc.percona.com/upgrade-haproxy patched perconaxtradbcluster.pxc.percona.com "upgrade-haproxy" deleted from upgrade-haproxy-21598 namespace perconaxtradbclusterrestore.pxc.percona.com "on-demand-backup-minio" deleted from upgrade-haproxy-21598 namespace validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "upgrade-haproxy-21598" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusterbackups.pxc.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusterrestores.pxc.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaxtradbclusters.pxc.percona.com" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------