Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2521/e2e-tests/logs/pvc-auto-resize-5-7.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Checking if default storageclass standard-rwo allows volume expansion No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster11" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0611 02:57:52.494766 20145 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1781146670247033000&timeoutSeconds=560&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0611 02:57:55.562790 20550 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1781146675247105000&timeoutSeconds=457&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0611 02:57:58.876990 21024 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1781146675247105000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-774dd4b69-zrv2d condition met E0611 02:58:24.304621 25020 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-zrv2d&resourceVersion=1781146703944833000&timeoutSeconds=414&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-774dd4b69-zrv2d condition met E0611 02:58:29.769932 25868 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-zrv2d&resourceVersion=1781146706965397000&timeoutSeconds=440&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-774dd4b69-zrv2d to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pvc-auto-resize-16784 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pvc-auto-resize-16784" not found waiting for namespace/pvc-auto-resize-16784 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pvc-auto-resize-16784" not found ----------------------------------------------------------------------------------- create namespace pvc-auto-resize-16784 ----------------------------------------------------------------------------------- namespace/pvc-auto-resize-16784 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster11" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with storage autoscaling enabled ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- waiting pods to be running for some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for some-name-pxc ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 2 running pods for some-name-haproxy ----------------------------------------------------------------------------------- pod/some-name-haproxy-0 condition met E0611 03:03:37.354272 8333 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-haproxy-0&resourceVersion=1781147012521465000&timeoutSeconds=378&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-haproxy-0 to become Ready.Ok pod/some-name-haproxy-1 condition met E0611 03:03:53.105017 10397 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-haproxy-1&resourceVersion=1781147032522040000&timeoutSeconds=301&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-haproxy-1 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-97cb9c68b-zfwm6 condition met E0611 03:04:48.695608 19562 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-97cb9c68b-zfwm6&resourceVersion=1781147087648236000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-97cb9c68b-zfwm6 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-97cb9c68b-zfwm6 condition met E0611 03:05:15.817069 23274 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-97cb9c68b-zfwm6&resourceVersion=1781147113148923000&timeoutSeconds=454&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-97cb9c68b-zfwm6 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-97cb9c68b-zfwm6 condition met E0611 03:06:00.192052 30222 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-97cb9c68b-zfwm6&resourceVersion=1781147158412721000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-97cb9c68b-zfwm6 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-97cb9c68b-zfwm6 condition met E0611 03:06:14.215332 32349 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-97cb9c68b-zfwm6&resourceVersion=1781147171879696000&timeoutSeconds=425&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-97cb9c68b-zfwm6 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-97cb9c68b-zfwm6 condition met E0611 03:06:26.852895 1719 reflector.go:227] "Failed to watch" err="Get \"https://34.30.129.139/api/v1/namespaces/pvc-auto-resize-16784/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-97cb9c68b-zfwm6&resourceVersion=1781147183194766000&timeoutSeconds=475&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-97cb9c68b-zfwm6 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify initial PVC size is 1Gi ----------------------------------------------------------------------------------- Initial PVC size: 1Gi ----------------------------------------------------------------------------------- fill disk to trigger autoscaling threshold 50 percent ----------------------------------------------------------------------------------- Filling disk on some-name-pxc-0 to approximately 50% (creating 600MB file) Current disk usage on some-name-pxc-0: 100% ----------------------------------------------------------------------------------- wait for auto-resize to trigger ----------------------------------------------------------------------------------- Waiting for PVC datadir-some-name-pxc-0 to auto-resize to 3Gi ... PVC datadir-some-name-pxc-0 successfully resized to 3Gi Waiting for PVC datadir-some-name-pxc-1 to auto-resize to 3Gi PVC datadir-some-name-pxc-1 successfully resized to 3Gi Waiting for PVC datadir-some-name-pxc-2 to auto-resize to 3Gi PVC datadir-some-name-pxc-2 successfully resized to 3Gi ----------------------------------------------------------------------------------- verify autoscaling status is updated ----------------------------------------------------------------------------------- Checking autoscaling status for PVCs: datadir-some-name-pxc-0 datadir-some-name-pxc-1 datadir-some-name-pxc-2 Expected resize count per PVC: 1 ----------------------------------------------------------------------------------- wait for cluster consistency after resize ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- verify cluster is ready ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test namespace resource quota enforcement ----------------------------------------------------------------------------------- Applying resourcequota for default storageclass standard-rwo with quota 9Gi resourcequota/default-storage-quota created Filling disk again to trigger autoscaling with quota limit Filling disk on some-name-pxc-0 to approximately 50% (creating 1700MB file) Current disk usage on some-name-pxc-0: 70% Waiting to verify PVC does not resize beyond quota (should stay at 3Gi) ----------------------------------------------------------------------------------- increase quota to allow further autoscaling ----------------------------------------------------------------------------------- Applying resourcequota for default storageclass standard-rwo with quota 15Gi resourcequota/default-storage-quota configured Waiting for PVC datadir-some-name-pxc-0 to auto-resize to 5Gi .... PVC datadir-some-name-pxc-0 successfully resized to 5Gi Waiting for PVC datadir-some-name-pxc-1 to auto-resize to 5Gi PVC datadir-some-name-pxc-1 successfully resized to 5Gi Waiting for PVC datadir-some-name-pxc-2 to auto-resize to 5Gi PVC datadir-some-name-pxc-2 successfully resized to 5Gi ----------------------------------------------------------------------------------- verify autoscaling status after quota test ----------------------------------------------------------------------------------- Checking autoscaling status for PVCs: datadir-some-name-pxc-0 datadir-some-name-pxc-1 datadir-some-name-pxc-2 Expected resize count per PVC: 2 ----------------------------------------------------------------------------------- verify cluster consistency after quota test ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- test maxSize limit ----------------------------------------------------------------------------------- Applying resourcequota for default storageclass standard-rwo with quota 30Gi resourcequota/default-storage-quota configured Continuing to fill disk to test maxSize enforcement Filling disk on some-name-pxc-0 to approximately 50% (creating 2900MB file) Current disk usage on some-name-pxc-0: 66% Waiting for PVC datadir-some-name-pxc-0 to auto-resize to 7Gi ....... PVC datadir-some-name-pxc-0 successfully resized to 7Gi Waiting for PVC datadir-some-name-pxc-1 to auto-resize to 7Gi PVC datadir-some-name-pxc-1 successfully resized to 7Gi Waiting for PVC datadir-some-name-pxc-2 to auto-resize to 7Gi PVC datadir-some-name-pxc-2 successfully resized to 7Gi Filling disk on some-name-pxc-0 to approximately 50% (creating 4000MB file) Current disk usage on some-name-pxc-0: 63% Waiting for PVC datadir-some-name-pxc-0 to auto-resize to 9Gi ... PVC datadir-some-name-pxc-0 successfully resized to 9Gi Waiting for PVC datadir-some-name-pxc-1 to auto-resize to 9Gi PVC datadir-some-name-pxc-1 successfully resized to 9Gi Waiting for PVC datadir-some-name-pxc-2 to auto-resize to 9Gi PVC datadir-some-name-pxc-2 successfully resized to 9Gi Filling disk on some-name-pxc-0 to approximately 50% (creating 5200MB file) Current disk usage on some-name-pxc-0: 62% PVC is at 9Gi, will resize to 10Gi (maxSize) Waiting for PVC datadir-some-name-pxc-0 to auto-resize to 10Gi . PVC datadir-some-name-pxc-0 successfully resized to 10Gi Waiting for PVC datadir-some-name-pxc-1 to auto-resize to 10Gi ... PVC datadir-some-name-pxc-1 successfully resized to 10Gi Waiting for PVC datadir-some-name-pxc-2 to auto-resize to 10Gi PVC datadir-some-name-pxc-2 successfully resized to 10Gi ----------------------------------------------------------------------------------- verify final autoscaling status ----------------------------------------------------------------------------------- Checking autoscaling status for PVCs: datadir-some-name-pxc-0 datadir-some-name-pxc-1 datadir-some-name-pxc-2 Expected resize count per PVC: 5 ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched perconaxtradbcluster.pxc.percona.com "some-name" deleted from pvc-auto-resize-16784 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "pvc-auto-resize-16784" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------