Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1932/e2e-tests/logs/demand-backup-incremental.log WARNING: version difference between client (1.33) and server (1.30) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.33) and server (1.30) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.33) and server (1.30) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- namespace "gke-managed-cim" deleted namespace "gke-managed-system" deleted ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace "gmp-public" deleted namespace "gmp-system" deleted namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1932-138a34ea-1-cluster1" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-78748cc64c-g9qn2 to be ready.OK ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-incremental-26277 ----------------------------------------------------------------------------------- namespace "gke-managed-cim" deleted namespace "gke-managed-system" deleted namespace "gmp-public" deleted namespace "gmp-system" deleted ----------------------------------------------------------------------------------- create namespace demand-backup-incremental-26277 ----------------------------------------------------------------------------------- namespace/demand-backup-incremental-26277 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1932-138a34ea-1-cluster1" modified. ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found "minio" has been removed from your repositories "minio" has been added to your repositories NAME: minio-service LAST DEPLOYED: Mon May 19 01:44:35 2025 NAMESPACE: demand-backup-incremental-26277 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.demand-backup-incremental-26277.svc.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace demand-backup-incremental-26277 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace demand-backup-incremental-26277 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace demand-backup-incremental-26277 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace demand-backup-incremental-26277 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local waiting for pod/minio-service-8967c7f7f-vhjcm to be ready.OK service/minio-service created make_bucket: operator-testing pod "aws-cli" deleted If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: Internal error occurred: unable to upgrade connection: container aws-cli not found in pod aws-cli_demand-backup-incremental-26277 ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- Testing on not sharded cluster ----------------------------------------------------------------------------------- [2025-05-19T01:45:20+0000] Creating PSMDB cluster secret/some-users created perconaservermongodb.psmdb.percona.com/some-name created deployment.apps/psmdb-client created [2025-05-19T01:45:25+0000] check if all pods started waiting for pod/some-name-rs0-0 to be ready.........OK waiting for pod/some-name-rs0-1 to be ready.........OK waiting for pod/some-name-rs0-2 to be ready.......OK Waiting for cluster readyness. waiting for cluster readyness [2025-05-19T01:47:03+0000] writing test data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("b274d8c3-6831-47ec-9106-c0b2fde76288") } Percona Server for MongoDB server version: v8.0.8-3 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-0.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("51a35379-05fd-482e-bc7f-0603dc5f09db") } Percona Server for MongoDB server version: v8.0.8-3 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2025-05-19T01:47:15+0000] running db.test.find() in myApp [2025-05-19T01:47:18+0000] running db.test.find() in myApp [2025-05-19T01:47:22+0000] running db.test.find() in myApp [2025-05-19T01:47:27+0000] running backups perconaservermongodbbackup.psmdb.percona.com/backup-aws-s3 created perconaservermongodbbackup.psmdb.percona.com/backup-gcp-cs created perconaservermongodbbackup.psmdb.percona.com/backup-azure-blob created waiting for backup-aws-s3 to reach ready state...... [2025-05-19T01:47:55+0000] checking if https://s3.amazonaws.com/operator-testing/psmdb-demand-backup-incremental/2025-05-19T01:47:31Z/rs0/filelist.pbm exists % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 12363 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK x-amz-id-2: lSK1U+g+jU/toF7Xpo8hDj39uj5wMFaAZsCuObykayVonEuhx8CTLqeSBxbx86vnvIL3NYNJwV4= x-amz-request-id: 95F14TNJQ1K0RQ99 Date: Mon, 19 May 2025 01:47:56 GMT Last-Modified: Mon, 19 May 2025 01:47:49 GMT x-amz-expiration: expiry-date="Wed, 21 May 2025 00:00:00 GMT", rule-id="1 Days Cleanup" ETag: "149df17a636851fda6225882a98985d6" x-amz-server-side-encryption: AES256 Accept-Ranges: bytes Content-Type: binary/octet-stream Content-Length: 12363 Server: AmazonS3 waiting for backup-gcp-cs to reach ready state............ [2025-05-19T01:48:27+0000] checking if https://storage.googleapis.com/operator-testing/psmdb-demand-backup-incremental/2025-05-19T01:48:09Z/rs0/filelist.pbm exists % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 12363 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 12363 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/2 200 content-type: application/octet-stream x-guploader-uploadid: AAO2VwrE-Z1rU84HXzgiYfEgXwCagzoL-3DDSPn35P2PC9ttYOe-CQwov5g3Y2sWNGBi7BAZlMUNgEw expires: Mon, 19 May 2025 02:48:28 GMT date: Mon, 19 May 2025 01:48:28 GMT cache-control: public, max-age=3600 last-modified: Mon, 19 May 2025 01:48:23 GMT etag: "f4a5ac371f4786477556a241f82ea141" x-goog-generation: 1747619303362792 x-goog-metageneration: 1 x-goog-stored-content-encoding: identity x-goog-stored-content-length: 12363 x-goog-hash: crc32c=7B9Ktg== x-goog-hash: md5=9KWsNx9Hhkd1VqJB+C6hQQ== x-goog-expiration: Tue, 20 May 2025 01:48:23 GMT x-goog-storage-class: STANDARD accept-ranges: bytes content-length: 12363 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 waiting for backup-azure-blob to reach ready state. [2025-05-19T01:48:32+0000] checking if https://engk8soperators.blob.core.windows.net/operator-testing/psmdb-demand-backup-incremental/2025-05-19T01:47:53Z/rs0/filelist.pbm exists % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 12363 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Content-Length: 12363 Content-Type: application/octet-stream Content-MD5: ik7Hvh/Z71QfRkSpG41IZA== Last-Modified: Mon, 19 May 2025 01:48:05 GMT ETag: 0x8DD96773320653C Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: e71381d8-c01e-0074-1360-c8eb9f000000 x-ms-version: 2009-09-19 x-ms-lease-status: unlocked x-ms-blob-type: BlockBlob Date: Mon, 19 May 2025 01:48:31 GMT perconaservermongodbbackup.psmdb.percona.com/backup-minio created waiting for backup-minio to reach ready state..... Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("428fdf74-5721-413b-a014-e6d614f2d5b9") } Percona Server for MongoDB server version: v8.0.8-3 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2025-05-19T01:48:54+0000] running db.test.find() in myApp [2025-05-19T01:48:58+0000] running db.test.find() in myApp [2025-05-19T01:49:00+0000] running db.test.find() in myApp perconaservermongodbbackup.psmdb.percona.com/backup-minio-not-base created waiting for backup-minio-not-base to reach ready state..... [2025-05-19T01:49:20+0000] drop collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-26277.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("2f8cde19-d513-4250-a9c3-6f6ae32f334f") } Percona Server for MongoDB server version: v8.0.8-3 WARNING: shell and server versions do not match switched to db myApp true bye [2025-05-19T01:49:22+0000] check backup and restore -- backup-aws-s3 perconaservermongodbrestore.psmdb.percona.com/restore-backup-aws-s3 created waiting psmdb-restore/restore-backup-aws-s3 to reach requested state...............................................................error: Get "https://34.10.249.112/apis/psmdb.percona.com/v1/namespaces/demand-backup-incremental-26277/perconaservermongodbrestores/restore-backup-aws-s3": dial tcp 34.10.249.112:443: i/o timeout - error from a previous attempt: http2: client connection lost Unable to connect to the server: dial tcp 34.10.249.112:443: i/o timeout Unable to connect to the server: dial tcp 34.10.249.112:443: i/o timeout Unable to connect to the server: dial tcp 34.10.249.112:443: i/o timeout