Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2476/e2e-tests/logs/extra-pvc-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2476-a8b01a39-5-cluster1" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-8548fd5788-hmgn9 condition met E0516 19:09:53.144871 24769 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-8548fd5788-hmgn9&resourceVersion=1778958592782441000&timeoutSeconds=479&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-8548fd5788-hmgn9 condition met E0516 19:09:57.226748 25382 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-8548fd5788-hmgn9&resourceVersion=1778958595137011000&timeoutSeconds=583&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-8548fd5788-hmgn9 to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces extra-pvc-2184 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "extra-pvc-2184" not found waiting for namespace/extra-pvc-2184 to be deletedError from server (NotFound): namespaces "extra-pvc-2184" not found ----------------------------------------------------------------------------------- create namespace extra-pvc-2184 ----------------------------------------------------------------------------------- namespace/extra-pvc-2184 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2476-a8b01a39-5-cluster1" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create extra PVCs ----------------------------------------------------------------------------------- [2026-05-16T19:10:35+0000] Creating extra PVCs in namespace extra-pvc-2184 persistentvolumeclaim/extra-storage-source created pod/pvc-populator created persistentvolumeclaim/extra-storage-0 created persistentvolumeclaim/extra-storage-1 created persistentvolumeclaim/extra-storage-2 created [2026-05-16T19:10:40+0000] Waiting for source PVC to be bound persistentvolumeclaim/extra-storage-source condition met E0516 19:10:43.309206 32112 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/persistentvolumeclaims?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextra-storage-source&resourceVersion=1778958642722367011&timeoutSeconds=519&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" [2026-05-16T19:10:43+0000] Waiting for populator pod to be ready pod/pvc-populator condition met ----------------------------------------------------------------------------------- create PXC cluster without extra PVCs ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/some-name-haproxy-0 condition met waiting for pod/some-name-haproxy-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met E0516 19:12:04.394903 11293 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-pxc-0&resourceVersion=1778958723106687010&timeoutSeconds=469&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:14:52.444716 3753 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778958891420710000&timeoutSeconds=560&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:15:03.615365 5331 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778958901670589000&timeoutSeconds=444&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:15:43.363661 11171 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778958941456594000&timeoutSeconds=326&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:15:55.880582 13021 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778958953800677000&timeoutSeconds=312&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:16:10.111941 15203 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778958967794392000&timeoutSeconds=380&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify cluster is running without extra PVCs ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- test adding extra PVC to running cluster ----------------------------------------------------------------------------------- [2026-05-16T19:16:37+0000] Patching cluster some-name to add extraPVCs perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready....................... ----------------------------------------------------------------------------------- verify extra PVC is configured in StatefulSet ----------------------------------------------------------------------------------- [2026-05-16T19:19:54+0000] Verifying extra PVC configuration in StatefulSet some-name-pxc [2026-05-16T19:19:57+0000] Volume extra-data-volume with claimName extra-storage-0 found in StatefulSet [2026-05-16T19:19:59+0000] Volume mount extra-data-volume with mountPath /var/lib/mysql-extra found in pxc container ----------------------------------------------------------------------------------- verify extra PVC is mounted in all PXC pods ----------------------------------------------------------------------------------- [2026-05-16T19:19:59+0000] Verifying extra PVC is mounted in pod some-name-pxc-0 at /var/lib/mysql-extra [2026-05-16T19:20:01+0000] Mount path /var/lib/mysql-extra exists in pod some-name-pxc-0 [2026-05-16T19:20:01+0000] Verifying extra PVC is mounted in pod some-name-pxc-1 at /var/lib/mysql-extra [2026-05-16T19:20:03+0000] Mount path /var/lib/mysql-extra exists in pod some-name-pxc-1 [2026-05-16T19:20:03+0000] Verifying extra PVC is mounted in pod some-name-pxc-2 at /var/lib/mysql-extra [2026-05-16T19:20:05+0000] Mount path /var/lib/mysql-extra exists in pod some-name-pxc-2 ----------------------------------------------------------------------------------- test adding multiple extra PVCs ----------------------------------------------------------------------------------- [2026-05-16T19:20:05+0000] Patching cluster some-name to add multiple extraPVCs perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.................................. ----------------------------------------------------------------------------------- verify multiple extra PVCs are mounted ----------------------------------------------------------------------------------- [2026-05-16T19:24:15+0000] Verifying extra PVC is mounted in pod some-name-pxc-0 at /var/lib/mysql-extra-1 [2026-05-16T19:24:17+0000] Mount path /var/lib/mysql-extra-1 exists in pod some-name-pxc-0 [2026-05-16T19:24:19+0000] Read-only mount /var/lib/mysql-extra-2 is properly read-only in pod some-name-pxc-0 [2026-05-16T19:24:19+0000] Verifying extra PVC is mounted in pod some-name-pxc-1 at /var/lib/mysql-extra-1 [2026-05-16T19:24:22+0000] Mount path /var/lib/mysql-extra-1 exists in pod some-name-pxc-1 [2026-05-16T19:24:24+0000] Read-only mount /var/lib/mysql-extra-2 is properly read-only in pod some-name-pxc-0 [2026-05-16T19:24:24+0000] Verifying extra PVC is mounted in pod some-name-pxc-2 at /var/lib/mysql-extra-1 [2026-05-16T19:24:25+0000] Mount path /var/lib/mysql-extra-1 exists in pod some-name-pxc-2 [2026-05-16T19:24:27+0000] Read-only mount /var/lib/mysql-extra-2 is properly read-only in pod some-name-pxc-0 ----------------------------------------------------------------------------------- test removing extra PVCs from cluster ----------------------------------------------------------------------------------- [2026-05-16T19:24:27+0000] Patching cluster some-name to remove extraPVCs perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready........................ ----------------------------------------------------------------------------------- verify extra PVCs are removed from StatefulSet ----------------------------------------------------------------------------------- [2026-05-16T19:27:35+0000] Extra PVCs successfully removed from StatefulSet ----------------------------------------------------------------------------------- Deleting some-name and recreating PXC cluster ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com "some-name" deleted from extra-pvc-2184 namespace ----------------------------------------------------------------------------------- test creating cluster with extra PVCs from start ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl unchanged secret/some-name-ssl-internal unchanged deployment.apps/pxc-client unchanged perconaxtradbcluster.pxc.percona.com/some-name-2 created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- Error from server (NotFound): pods "some-name-2-haproxy-0" not found waiting for pod/some-name-2-haproxy-0 to become Ready.....................................Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/some-name-2-pxc-0 condition met E0516 19:29:34.113425 22556 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-2-pxc-0&resourceVersion=1778959773820927019&timeoutSeconds=524&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-2-pxc-0 to become Ready.Ok pod/some-name-2-pxc-1 condition met waiting for pod/some-name-2-pxc-1 to become Ready.Ok pod/some-name-2-pxc-2 condition met waiting for pod/some-name-2-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:32:24.255369 12527 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778959942651703000&timeoutSeconds=311&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:32:31.621251 13712 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778959950640508000&timeoutSeconds=541&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:33:09.549055 18570 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778959987767326000&timeoutSeconds=347&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:33:18.957206 19874 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778959997417490000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-m2jrq condition met E0516 19:33:27.999708 20988 reflector.go:227] "Failed to watch" err="Get \"https://35.254.186.129/api/v1/namespaces/extra-pvc-2184/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-m2jrq&resourceVersion=1778960006392107000&timeoutSeconds=545&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-m2jrq to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name-2 to be ready ----------------------------------------------------------------------------------- verify extra PVC is configured in new cluster ----------------------------------------------------------------------------------- [2026-05-16T19:33:45+0000] Verifying extra PVC configuration in StatefulSet some-name-2-pxc [2026-05-16T19:33:47+0000] Volume extra-data-volume with claimName extra-storage-0 found in StatefulSet [2026-05-16T19:33:48+0000] Volume mount extra-data-volume with mountPath /var/lib/mysql-extra found in pxc container [2026-05-16T19:33:48+0000] Verifying extra PVC is mounted in pod some-name-2-pxc-0 at /var/lib/mysql-extra [2026-05-16T19:33:51+0000] Mount path /var/lib/mysql-extra exists in pod some-name-2-pxc-0 [2026-05-16T19:33:51+0000] Verifying extra PVC is mounted in pod some-name-2-pxc-1 at /var/lib/mysql-extra [2026-05-16T19:33:53+0000] Mount path /var/lib/mysql-extra exists in pod some-name-2-pxc-1 [2026-05-16T19:33:53+0000] Verifying extra PVC is mounted in pod some-name-2-pxc-2 at /var/lib/mysql-extra [2026-05-16T19:33:55+0000] Mount path /var/lib/mysql-extra exists in pod some-name-2-pxc-2 ----------------------------------------------------------------------------------- cleanup second cluster ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com "some-name-2" deleted from extra-pvc-2184 namespace ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------