Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/logs/scheduled-backup.log
WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1
WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1
WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1


-----------------------------------------------------------------------------------
get and delete old CRDs and RBAC
-----------------------------------------------------------------------------------

error: the server doesn't have a resource type "perconaservermongodbbackups"
+ kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}'
error: the server doesn't have a resource type "perconaservermongodbbackups"
error: the server doesn't have a resource type "perconaservermongodbrestores"
+ kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}'
error: the server doesn't have a resource type "perconaservermongodbrestores"
error: the server doesn't have a resource type "perconaservermongodbs"
+ kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}'
error: the server doesn't have a resource type "perconaservermongodbs"
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found


-----------------------------------------------------------------------------------
destroy chaos-mesh
-----------------------------------------------------------------------------------

error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified


-----------------------------------------------------------------------------------
cleaned up all old namespaces
-----------------------------------------------------------------------------------



-----------------------------------------------------------------------------------
cleaned up old namespaces psmdb-operator
-----------------------------------------------------------------------------------

error: resource(s) were provided, but no name was specified


-----------------------------------------------------------------------------------
create namespace psmdb-operator
-----------------------------------------------------------------------------------

namespace/psmdb-operator created
Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1546-9c491eef-1-cluster2" modified.


-----------------------------------------------------------------------------------
start PSMDB operator
-----------------------------------------------------------------------------------

customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied
clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created
serviceaccount/percona-server-mongodb-operator created
clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created
deployment.apps/percona-server-mongodb-operator created
waiting for pod/percona-server-mongodb-operator-65d8c47b97-qpk7f to be ready.OK


-----------------------------------------------------------------------------------
destroy chaos-mesh
-----------------------------------------------------------------------------------

error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified
error: resource(s) were provided, but no name was specified


-----------------------------------------------------------------------------------
cleaned up all old namespaces
-----------------------------------------------------------------------------------



-----------------------------------------------------------------------------------
cleaned up old namespaces scheduled-backup-16043
-----------------------------------------------------------------------------------

error: resource(s) were provided, but no name was specified


-----------------------------------------------------------------------------------
create namespace scheduled-backup-16043
-----------------------------------------------------------------------------------

namespace/scheduled-backup-16043 created
Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1546-9c491eef-1-cluster2" modified.


-----------------------------------------------------------------------------------
install Minio
-----------------------------------------------------------------------------------

Error: uninstall: Release not loaded: minio-service: release: not found
"minio" has been removed from your repositories
"minio" has been added to your repositories
NAME: minio-service
LAST DEPLOYED: Wed May  8 15:53:52 2024
NAMESPACE: scheduled-backup-16043
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MinIO can be accessed via port 9000 on the following DNS name from within your cluster:
minio-service.scheduled-backup-16043.svc.cluster.local

To access MinIO from localhost, run the below commands:

  1. export POD_NAME=$(kubectl get pods --namespace scheduled-backup-16043 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}")

  2. kubectl port-forward $POD_NAME 9000 --namespace scheduled-backup-16043

Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/

You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client:

  1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart

  2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace scheduled-backup-16043 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace scheduled-backup-16043 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000

  3. mc ls minio-service-local
waiting for pod/minio-service-57dd49b-lqx8s to be ready.OK
service/minio-service created
make_bucket: operator-testing
pod "aws-cli" deleted
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/aws-cli, falling back to streaming logs: unable to upgrade connection: container aws-cli not found in pod aws-cli_scheduled-backup-16043


-----------------------------------------------------------------------------------
add labels
-----------------------------------------------------------------------------------

node/gke-jen-psmdb-1546-9c491-default-pool-7a76c898-7q72 labeled


-----------------------------------------------------------------------------------
create PriorityClass
-----------------------------------------------------------------------------------

priorityclass.scheduling.k8s.io/high-priority created


-----------------------------------------------------------------------------------
create secrets and start client
-----------------------------------------------------------------------------------

secret/some-users created
deployment.apps/psmdb-client created


-----------------------------------------------------------------------------------
create secrets for cloud storages
-----------------------------------------------------------------------------------

secret/minio-secret created
secret/aws-s3-secret created
secret/gcp-cs-secret created
secret/azure-secret created


-----------------------------------------------------------------------------------
create first PSMDB cluster some-name-rs0
-----------------------------------------------------------------------------------

perconaservermongodb.psmdb.percona.com/some-name created


-----------------------------------------------------------------------------------
check if all 3 Pods started
-----------------------------------------------------------------------------------

waiting for pod/some-name-rs0-0 to be ready.......OK
waiting for pod/some-name-rs0-1 to be ready......OK
waiting for pod/some-name-rs0-2 to be ready......OK
Waiting for cluster readyness...


-----------------------------------------------------------------------------------
check if service and statefulset created with expected config
-----------------------------------------------------------------------------------



-----------------------------------------------------------------------------------
create user myApp
-----------------------------------------------------------------------------------

Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("bfa02ef3-fe36-40cd-8190-762a9258057e") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
Successfully added user: {
	"user" : "myApp",
	"roles" : [
		{
			"db" : "myApp",
			"role" : "readWrite"
		}
	]
}
bye


-----------------------------------------------------------------------------------
write data, read from all
-----------------------------------------------------------------------------------

Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("0a1bf213-3945-4b08-a4e6-d4cfdda4b4f8") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
switched to db myApp
WriteResult({ "nInserted" : 1 })
bye
some-name-rs0-0
some-name-rs0-1
some-name-rs0-2


-----------------------------------------------------------------------------------
add backups schedule, wait for the first backup
-----------------------------------------------------------------------------------

perconaservermongodb.psmdb.percona.com/some-name configured
perconaservermongodb.psmdb.percona.com/some-name configured
cron-some-name-20240508155700-27fxf.
cron-some-name-20240508155700-8kztq.............
cron-some-name-20240508155700-8rczl................
cron-some-name-20240508155700-cnpbb...............


-----------------------------------------------------------------------------------
check backup and restore -- minio
-----------------------------------------------------------------------------------

If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/aws-cli, falling back to streaming logs: unable to upgrade connection: container aws-cli not found in pod aws-cli_scheduled-backup-16043
2024-05-08 15:59:00         56 myApp.test.gz
Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("61959635-3e45-4a28-bdbe-d65ea173b028") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
switched to db myApp
WriteResult({ "nInserted" : 1 })
bye
perconaservermongodbrestore.psmdb.percona.com/restore-cron-some-name-20240508155700-cnpbb created
waiting psmdb-restore/cron-some-name-20240508155700-cnpbb to reach ready state...........
+ '[' 1 -eq 1 ']'
+ wait_cluster_consistency some-name
+ local cluster_name=some-name
+ local wait_time=32
+ retry=0
+ sleep 7
+ echo -n 'waiting for cluster readyness'
waiting for cluster readyness++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.JIzEXXaG0C
+++ mktemp
++ local LAST_ERR=/tmp/tmp.alwKtuYiSF
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.JIzEXXaG0C
++ cat /tmp/tmp.alwKtuYiSF
++ rm /tmp/tmp.JIzEXXaG0C /tmp/tmp.alwKtuYiSF
++ return 0
+ [[ ready == \r\e\a\d\y ]]
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.LwZCVCvk4X
+++ mktemp
++ local LAST_ERR=/tmp/tmp.UnmtwjHX0V
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.LwZCVCvk4X
++ cat /tmp/tmp.UnmtwjHX0V
++ rm /tmp/tmp.LwZCVCvk4X /tmp/tmp.UnmtwjHX0V
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.U4D0iWTfvZ
++ mktemp
+ local LAST_ERR=/tmp/tmp.6CLtjOS2Hz
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.U4D0iWTfvZ
+ cat /tmp/tmp.6CLtjOS2Hz
+ rm /tmp/tmp.U4D0iWTfvZ /tmp/tmp.6CLtjOS2Hz
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.OS3dNyPhHu
+++ mktemp
++ local LAST_ERR=/tmp/tmp.i01nob7Apx
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.OS3dNyPhHu
++ cat /tmp/tmp.i01nob7Apx
++ rm /tmp/tmp.OS3dNyPhHu /tmp/tmp.i01nob7Apx
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.AA4I5uK90N
++ mktemp
+ local LAST_ERR=/tmp/tmp.wTOUhNQInJ
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.AA4I5uK90N
+ cat /tmp/tmp.wTOUhNQInJ
+ rm /tmp/tmp.AA4I5uK90N /tmp/tmp.wTOUhNQInJ
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.SjoDke2Q1I
+++ mktemp
++ local LAST_ERR=/tmp/tmp.opRYVIarOQ
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.SjoDke2Q1I
++ cat /tmp/tmp.opRYVIarOQ
++ rm /tmp/tmp.SjoDke2Q1I /tmp/tmp.opRYVIarOQ
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.KwyLYKOYRZ
++ mktemp
+ local LAST_ERR=/tmp/tmp.dYrNBPA2f1
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.KwyLYKOYRZ
+ cat /tmp/tmp.dYrNBPA2f1
+ rm /tmp/tmp.KwyLYKOYRZ /tmp/tmp.dYrNBPA2f1
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ '[' -z '' ']'
+ desc 'check backup and restore -- aws-s3'
+ set +o xtrace


-----------------------------------------------------------------------------------
check backup and restore -- aws-s3
-----------------------------------------------------------------------------------

Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("d2a96e1b-d48c-4f2b-9786-9cff08930a41") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
switched to db myApp
WriteResult({ "nInserted" : 1 })
bye
perconaservermongodbrestore.psmdb.percona.com/restore-cron-some-name-20240508155700-27fxf created
waiting psmdb-restore/cron-some-name-20240508155700-27fxf to reach ready state..................
+ '[' 1 -eq 1 ']'
+ wait_cluster_consistency some-name
+ local cluster_name=some-name
+ local wait_time=32
+ retry=0
+ sleep 7
+ echo -n 'waiting for cluster readyness'
waiting for cluster readyness++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.SsxRhgyeis
+++ mktemp
++ local LAST_ERR=/tmp/tmp.5cSN1i99Gu
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.SsxRhgyeis
++ cat /tmp/tmp.5cSN1i99Gu
++ rm /tmp/tmp.SsxRhgyeis /tmp/tmp.5cSN1i99Gu
++ return 0
+ [[ ready == \r\e\a\d\y ]]
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 mongodb ''
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.YpY5WujyXy
+++ mktemp
++ local LAST_ERR=/tmp/tmp.HV0AbhGhxW
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.YpY5WujyXy
++ cat /tmp/tmp.HV0AbhGhxW
++ rm /tmp/tmp.YpY5WujyXy /tmp/tmp.HV0AbhGhxW
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.h95XGLSKus
++ mktemp
+ local LAST_ERR=/tmp/tmp.hnTM1qNga5
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.h95XGLSKus
+ cat /tmp/tmp.hnTM1qNga5
+ rm /tmp/tmp.h95XGLSKus /tmp/tmp.hnTM1qNga5
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 mongodb ''
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.POsc0YbbQX
+++ mktemp
++ local LAST_ERR=/tmp/tmp.o4rU7oIGkB
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.POsc0YbbQX
++ cat /tmp/tmp.o4rU7oIGkB
++ rm /tmp/tmp.POsc0YbbQX /tmp/tmp.o4rU7oIGkB
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.I3USDvRgBR
++ mktemp
+ local LAST_ERR=/tmp/tmp.pMiv4TvEYu
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.I3USDvRgBR
+ cat /tmp/tmp.pMiv4TvEYu
+ rm /tmp/tmp.I3USDvRgBR /tmp/tmp.pMiv4TvEYu
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 mongodb ''
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.ROxoCBjTWN
+++ mktemp
++ local LAST_ERR=/tmp/tmp.KgkJAefrTE
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.ROxoCBjTWN
++ cat /tmp/tmp.KgkJAefrTE
++ rm /tmp/tmp.ROxoCBjTWN /tmp/tmp.KgkJAefrTE
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.qau8a59VJ7
++ mktemp
+ local LAST_ERR=/tmp/tmp.s9Lh067Z0b
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.qau8a59VJ7
+ cat /tmp/tmp.s9Lh067Z0b
+ rm /tmp/tmp.qau8a59VJ7 /tmp/tmp.s9Lh067Z0b
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ desc 'check backup and restore -- gcp-cs'
+ set +o xtrace


-----------------------------------------------------------------------------------
check backup and restore -- gcp-cs
-----------------------------------------------------------------------------------

Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("0e0b719a-3597-4686-b29b-a0551f26c70b") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
switched to db myApp
WriteResult({ "nInserted" : 1 })
bye
perconaservermongodbrestore.psmdb.percona.com/restore-cron-some-name-20240508155700-8kztq created
waiting psmdb-restore/cron-some-name-20240508155700-8kztq to reach ready state..............
+ '[' 1 -eq 1 ']'
+ wait_cluster_consistency some-name
+ local cluster_name=some-name
+ local wait_time=32
+ retry=0
+ sleep 7
+ echo -n 'waiting for cluster readyness'
waiting for cluster readyness++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.IjQVlWI0Bo
+++ mktemp
++ local LAST_ERR=/tmp/tmp.oAcxpct93Y
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.IjQVlWI0Bo
++ cat /tmp/tmp.oAcxpct93Y
++ rm /tmp/tmp.IjQVlWI0Bo /tmp/tmp.oAcxpct93Y
++ return 0
+ [[ ready == \r\e\a\d\y ]]
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.rJLINS9ZIE
+++ mktemp
++ local LAST_ERR=/tmp/tmp.4aN0GKB8e9
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.rJLINS9ZIE
++ cat /tmp/tmp.4aN0GKB8e9
++ rm /tmp/tmp.rJLINS9ZIE /tmp/tmp.4aN0GKB8e9
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.FA6dFhxJ51
++ mktemp
+ local LAST_ERR=/tmp/tmp.BadvXn6HpI
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.FA6dFhxJ51
+ cat /tmp/tmp.BadvXn6HpI
+ rm /tmp/tmp.FA6dFhxJ51 /tmp/tmp.BadvXn6HpI
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.05TGWwsTEw
+++ mktemp
++ local LAST_ERR=/tmp/tmp.y7cuADT3HL
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.05TGWwsTEw
++ cat /tmp/tmp.y7cuADT3HL
++ rm /tmp/tmp.05TGWwsTEw /tmp/tmp.y7cuADT3HL
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.82fHzEfb2L
++ mktemp
+ local LAST_ERR=/tmp/tmp.cfCzvrSUEb
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.82fHzEfb2L
+ cat /tmp/tmp.cfCzvrSUEb
+ rm /tmp/tmp.82fHzEfb2L /tmp/tmp.cfCzvrSUEb
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 mongodb ''
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.GhYHS8CVgh
+++ mktemp
++ local LAST_ERR=/tmp/tmp.l0g0JT98qT
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.GhYHS8CVgh
++ cat /tmp/tmp.l0g0JT98qT
++ rm /tmp/tmp.GhYHS8CVgh /tmp/tmp.l0g0JT98qT
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.eg5xdh8Cwl
++ mktemp
+ local LAST_ERR=/tmp/tmp.a03cQW0fe6
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.eg5xdh8Cwl
+ cat /tmp/tmp.a03cQW0fe6
+ rm /tmp/tmp.eg5xdh8Cwl /tmp/tmp.a03cQW0fe6
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ desc 'check backup and restore -- azure-blob'
+ set +o xtrace


-----------------------------------------------------------------------------------
check backup and restore -- azure-blob
-----------------------------------------------------------------------------------

Percona Server for MongoDB shell version v4.4.29-28
connecting to: mongodb://some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false
Implicit session: session { "id" : UUID("104c0344-12c2-4761-9ed1-b4cdb2947d3d") }
Percona Server for MongoDB server version: v7.0.8-5
WARNING: shell and server versions do not match
switched to db myApp
WriteResult({ "nInserted" : 1 })
bye
perconaservermongodbrestore.psmdb.percona.com/restore-cron-some-name-20240508155700-8rczl created
waiting psmdb-restore/cron-some-name-20240508155700-8rczl to reach ready state.............
+ '[' 1 -eq 1 ']'
+ wait_cluster_consistency some-name
+ local cluster_name=some-name
+ local wait_time=32
+ retry=0
+ sleep 7
+ echo -n 'waiting for cluster readyness'
waiting for cluster readyness++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.L3eBQCZn4O
+++ mktemp
++ local LAST_ERR=/tmp/tmp.9jhbzPuIBs
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.L3eBQCZn4O
++ cat /tmp/tmp.9jhbzPuIBs
++ rm /tmp/tmp.L3eBQCZn4O /tmp/tmp.9jhbzPuIBs
++ return 0
+ [[ ready == \r\e\a\d\y ]]
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.uLpjtgc1u7
+++ mktemp
++ local LAST_ERR=/tmp/tmp.WhJABEo2yu
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.uLpjtgc1u7
++ cat /tmp/tmp.WhJABEo2yu
++ rm /tmp/tmp.uLpjtgc1u7 /tmp/tmp.WhJABEo2yu
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.7U7TCAIFlq
++ mktemp
+ local LAST_ERR=/tmp/tmp.LIViJBGQ88
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.7U7TCAIFlq
+ cat /tmp/tmp.LIViJBGQ88
+ rm /tmp/tmp.7U7TCAIFlq /tmp/tmp.LIViJBGQ88
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.nH1Mz5QnbW
+++ mktemp
++ local LAST_ERR=/tmp/tmp.L6wjFPUOVN
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.nH1Mz5QnbW
++ cat /tmp/tmp.L6wjFPUOVN
++ rm /tmp/tmp.nH1Mz5QnbW /tmp/tmp.L6wjFPUOVN
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.FV9RUbc974
++ mktemp
+ local LAST_ERR=/tmp/tmp.mKF1UyCu39
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.FV9RUbc974
+ cat /tmp/tmp.mKF1UyCu39
+ rm /tmp/tmp.FV9RUbc974 /tmp/tmp.mKF1UyCu39
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.cHc6nCCiHG
+++ mktemp
++ local LAST_ERR=/tmp/tmp.9txYi6UaC0
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.cHc6nCCiHG
++ cat /tmp/tmp.9txYi6UaC0
++ rm /tmp/tmp.cHc6nCCiHG /tmp/tmp.9txYi6UaC0
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.ExR61UYRgM
++ mktemp
+ local LAST_ERR=/tmp/tmp.ait0EmliIo
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.ExR61UYRgM
+ cat /tmp/tmp.ait0EmliIo
+ rm /tmp/tmp.ExR61UYRgM /tmp/tmp.ait0EmliIo
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ desc 'add physical backup schedule, wait for the first backup'
+ set +o xtrace


-----------------------------------------------------------------------------------
add physical backup schedule, wait for the first backup
-----------------------------------------------------------------------------------

perconaservermongodb.psmdb.percona.com/some-name configured
perconaservermongodb.psmdb.percona.com/some-name configured
cron-some-name-20240508160400-mrq22......


-----------------------------------------------------------------------------------
check backup and restore -- minio
-----------------------------------------------------------------------------------

perconaservermongodbrestore.psmdb.percona.com/restore-cron-some-name-20240508160400-mrq22 created
waiting psmdb-restore/cron-some-name-20240508160400-mrq22 to reach ready state......................................................................................................
+ '[' 1 -eq 1 ']'
+ wait_cluster_consistency some-name
+ local cluster_name=some-name
+ local wait_time=32
+ retry=0
+ sleep 7
+ echo -n 'waiting for cluster readyness'
waiting for cluster readyness++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.HL5sBkN7iQ
+++ mktemp
++ local LAST_ERR=/tmp/tmp.csofJtF15W
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.HL5sBkN7iQ
++ cat /tmp/tmp.csofJtF15W
++ rm /tmp/tmp.HL5sBkN7iQ /tmp/tmp.csofJtF15W
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 1 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.6V1fs4RMuu
+++ mktemp
++ local LAST_ERR=/tmp/tmp.Y9lu2ntJER
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.6V1fs4RMuu
++ cat /tmp/tmp.Y9lu2ntJER
++ rm /tmp/tmp.6V1fs4RMuu /tmp/tmp.Y9lu2ntJER
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 2 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.bwecIvhjF0
+++ mktemp
++ local LAST_ERR=/tmp/tmp.hWND0FIheK
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.bwecIvhjF0
++ cat /tmp/tmp.hWND0FIheK
++ rm /tmp/tmp.bwecIvhjF0 /tmp/tmp.hWND0FIheK
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 3 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.UVK1rKen4I
+++ mktemp
++ local LAST_ERR=/tmp/tmp.yvkHrXtdZM
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.UVK1rKen4I
++ cat /tmp/tmp.yvkHrXtdZM
++ rm /tmp/tmp.UVK1rKen4I /tmp/tmp.yvkHrXtdZM
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 4 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.uMsmwXw0rU
+++ mktemp
++ local LAST_ERR=/tmp/tmp.bIQ4LFlMxY
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.uMsmwXw0rU
++ cat /tmp/tmp.bIQ4LFlMxY
++ rm /tmp/tmp.uMsmwXw0rU /tmp/tmp.bIQ4LFlMxY
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 5 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.8nVmVbP6Cu
+++ mktemp
++ local LAST_ERR=/tmp/tmp.7Zzm79o5AS
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.8nVmVbP6Cu
++ cat /tmp/tmp.7Zzm79o5AS
++ rm /tmp/tmp.8nVmVbP6Cu /tmp/tmp.7Zzm79o5AS
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 6 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.jMw8djVLnm
+++ mktemp
++ local LAST_ERR=/tmp/tmp.NID7j4ykSW
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.jMw8djVLnm
++ cat /tmp/tmp.NID7j4ykSW
++ rm /tmp/tmp.jMw8djVLnm /tmp/tmp.NID7j4ykSW
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 7 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.i84s1I9AAi
+++ mktemp
++ local LAST_ERR=/tmp/tmp.wHFTuO4cbT
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.i84s1I9AAi
++ cat /tmp/tmp.wHFTuO4cbT
++ rm /tmp/tmp.i84s1I9AAi /tmp/tmp.wHFTuO4cbT
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 8 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.oGkl7WbueS
+++ mktemp
++ local LAST_ERR=/tmp/tmp.cfmcJp8NOf
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.oGkl7WbueS
++ cat /tmp/tmp.cfmcJp8NOf
++ rm /tmp/tmp.oGkl7WbueS /tmp/tmp.cfmcJp8NOf
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 9 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.xpAsPzANUo
+++ mktemp
++ local LAST_ERR=/tmp/tmp.iXVGe1IgmX
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.xpAsPzANUo
++ cat /tmp/tmp.iXVGe1IgmX
++ rm /tmp/tmp.xpAsPzANUo /tmp/tmp.iXVGe1IgmX
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 10 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.Wc6MmGFFMG
+++ mktemp
++ local LAST_ERR=/tmp/tmp.niVKEkRJT2
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.Wc6MmGFFMG
++ cat /tmp/tmp.niVKEkRJT2
++ rm /tmp/tmp.Wc6MmGFFMG /tmp/tmp.niVKEkRJT2
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 11 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.noIOwGE7zt
+++ mktemp
++ local LAST_ERR=/tmp/tmp.AkfNnL0u8i
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.noIOwGE7zt
++ cat /tmp/tmp.AkfNnL0u8i
++ rm /tmp/tmp.noIOwGE7zt /tmp/tmp.AkfNnL0u8i
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 12 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.v46Tuft3tt
+++ mktemp
++ local LAST_ERR=/tmp/tmp.oo0fnAvork
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.v46Tuft3tt
++ cat /tmp/tmp.oo0fnAvork
++ rm /tmp/tmp.v46Tuft3tt /tmp/tmp.oo0fnAvork
++ return 0
+ [[ initializing == \r\e\a\d\y ]]
+ let retry+=1
+ '[' 13 -ge 32 ']'
+ echo -n .
.+ sleep 10
++ kubectl_bin get psmdb some-name -o 'jsonpath={.status.state}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.euvSc2Z1tt
+++ mktemp
++ local LAST_ERR=/tmp/tmp.Q8gG5HbGgZ
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get psmdb some-name -o 'jsonpath={.status.state}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.euvSc2Z1tt
++ cat /tmp/tmp.Q8gG5HbGgZ
++ rm /tmp/tmp.euvSc2Z1tt /tmp/tmp.Q8gG5HbGgZ
++ return 0
+ [[ ready == \r\e\a\d\y ]]
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ local LAST_OUT=/tmp/tmp.iMay9iTlBh
+++ mktemp
++ local LAST_ERR=/tmp/tmp.8iJjzdumYt
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.iMay9iTlBh
++ cat /tmp/tmp.8iJjzdumYt
++ rm /tmp/tmp.iMay9iTlBh /tmp/tmp.8iJjzdumYt
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.mqMQy8seUT
++ mktemp
+ local LAST_ERR=/tmp/tmp.uigMyDeraF
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-0.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.mqMQy8seUT
+ cat /tmp/tmp.uigMyDeraF
+ rm /tmp/tmp.mqMQy8seUT /tmp/tmp.uigMyDeraF
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 mongodb ''
+ local 'command=use myApp\n db.test.find()'
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local uri=myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.mWPbGdISaW
+++ mktemp
++ local LAST_ERR=/tmp/tmp.63MzVqHMc6
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.mWPbGdISaW
++ cat /tmp/tmp.63MzVqHMc6
++ rm /tmp/tmp.mWPbGdISaW /tmp/tmp.63MzVqHMc6
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.FNdmUjqfvv
++ mktemp
+ local LAST_ERR=/tmp/tmp.YWZKQf9bGE
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-1.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.FNdmUjqfvv
+ cat /tmp/tmp.YWZKQf9bGE
+ rm /tmp/tmp.FNdmUjqfvv /tmp/tmp.YWZKQf9bGE
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ compare_mongo_cmd find myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local command=find
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local postfix=
+ local suffix=
+ local database=myApp
+ local collection=test
+ run_mongo 'use myApp\n db.test.find()' myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 mongodb ''
+ egrep -v 'I NETWORK|W NETWORK|F NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:'
+ local 'command=use myApp\n db.test.find()'
+ local uri=myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043
+ local driver=mongodb
+ local suffix=.svc.cluster.local
+ /usr/bin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/'
++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
+++ mktemp
++ local LAST_OUT=/tmp/tmp.vuI2cGKouD
+++ mktemp
++ local LAST_ERR=/tmp/tmp.tEZ2qLIHZu
++ local exit_status=0
++ local timeout=4
+++ seq 0 2
++ for i in '$(seq 0 2)'
++ set +e
++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}'
++ exit_status=0
++ set -e
++ '[' 0 '!=' 0 -a -n 0 ']'
++ break
++ cat /tmp/tmp.vuI2cGKouD
++ cat /tmp/tmp.tEZ2qLIHZu
++ rm /tmp/tmp.vuI2cGKouD /tmp/tmp.tEZ2qLIHZu
++ return 0
+ local client_container=psmdb-client-7469665986-jjp2d
+ local mongo_flag=
+ [[ myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043 == *cfg* ]]
+ replica_set=rs0
+ kubectl_bin exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
++ mktemp
+ local LAST_OUT=/tmp/tmp.rqB3tVOgLs
++ mktemp
+ local LAST_ERR=/tmp/tmp.yPx5ahE8Tt
+ local exit_status=0
+ local timeout=4
++ seq 0 2
+ for i in '$(seq 0 2)'
+ set +e
+ kubectl exec psmdb-client-7469665986-jjp2d -- bash -c 'printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-rs0-2.some-name-rs0.scheduled-backup-16043.svc.cluster.local/admin?ssl=false\&replicaSet=rs0 '
+ exit_status=0
+ set -e
+ '[' 0 '!=' 0 -a -n 0 ']'
+ break
+ cat /tmp/tmp.rqB3tVOgLs
+ cat /tmp/tmp.yPx5ahE8Tt
+ rm /tmp/tmp.rqB3tVOgLs /tmp/tmp.yPx5ahE8Tt
+ return 0
+ diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1546/e2e-tests/scheduled-backup/compare/find.json /tmp/tmp.NZOHtUJ5tw/find
+ sleep 60
+ unlabel_node
+ desc 'remove labels'
+ set +o xtrace


-----------------------------------------------------------------------------------
remove labels
-----------------------------------------------------------------------------------

node/gke-jen-psmdb-1546-9c491-default-pool-7a76c898-7q72 labeled


-----------------------------------------------------------------------------------
destroy cluster/operator and all other resources
-----------------------------------------------------------------------------------



-----------------------------------------------------------------------------------
get and delete old CRDs and RBAC
-----------------------------------------------------------------------------------

customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted
customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted
customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted
+ kubectl patch perconaservermongodbbackups.psmdb.percona.com -n scheduled-backup-16043 cron-some-name-20240508155700-cnpbb --type=merge -p '{"metadata":{"finalizers":[]}}'
perconaservermongodbbackup.psmdb.percona.com/cron-some-name-20240508155700-cnpbb patched
+ kubectl patch perconaservermongodbbackups.psmdb.percona.com -n scheduled-backup-16043 cron-some-name-20240508160400-mrq22 --type=merge -p '{"metadata":{"finalizers":[]}}'
perconaservermongodbbackup.psmdb.percona.com/cron-some-name-20240508160400-mrq22 patched
customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com condition met
error: the server doesn't have a resource type "perconaservermongodbrestores"
+ kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}'
error: the server doesn't have a resource type "perconaservermongodbrestores"
error: the server doesn't have a resource type "perconaservermongodbs"
+ kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}'
error: the server doesn't have a resource type "perconaservermongodbs"
clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted


-----------------------------------------------------------------------------------
test passed
-----------------------------------------------------------------------------------