============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0 -- /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-7.0.8-100.fc43.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'html': '4.2.0', 'metadata': '3.1.1', 'dependency': '0.6.1', 'json-report': '1.5.0'}, 'CI': 'true', 'BUILD_NUMBER': '22', 'BUILD_ID': '22', 'BUILD_URL': 'https://cloud.cd.percona.com/job/cloud-psmdb-operator/job/PR-2058/22/', 'NODE_NAME': 'hcloud-3hmxjv56rzynxord', 'JOB_NAME': 'cloud-psmdb-operator/PR-2058', 'BUILD_TAG': 'jenkins-cloud-psmdb-operator-PR-2058-22', 'EXECUTOR_NUMBER': '0', 'JENKINS_URL': 'https://cloud.cd.percona.com/', 'WORKSPACE': '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058', 'GIT_COMMIT': '78121c56762758785b75e9b7c017e15fc887ae5d', 'GIT_URL': 'https://github.com/percona/percona-server-mongodb-operator.git', 'GIT_BRANCH': 'PR-2058'} rootdir: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058 configfile: pyproject.toml plugins: html-4.2.0, metadata-3.1.1, dependency-0.6.1, json-report-1.5.0 collecting ... collected 1 item e2e-tests/test_pytest_wrapper.py::test_bash_wrapper[demand-backup-sharded] [00:49:51.444884] WARNING kubectl error: Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 [00:49:53.290288] INFO Environment variables: KUBE_VERSION=1.33 EKS=0 GKE=1 OPENSHIFT= MINIKUBE= API=psmdb.percona.com/v1 REGISTRY_NAME=docker.io GIT_COMMIT=78121c56762758785b75e9b7c017e15fc887ae5d GIT_BRANCH=PR-2058 OPERATOR_VERSION=1.23.0 IMAGE=docker.io/perconalab/percona-server-mongodb-ope rator:PR-2058-78121c567 IMAGE_MONGOD=docker.io/perconalab/percona-server-mong odb-operator:main-mongod8.0 IMAGE_MONGOD_CHAIN=docker.io/perconalab/percona-serve r-mongodb-operator:main-mongod6.0 docker.io/perconalab/percona-server-mongodb-operator: main-mongod7.0 docker.io/perconalab/percona-server-mongodb-operator: main-mongod8.0 IMAGE_SEARCH=perconalab/percona-server-mongodb-operat or:main-mongot IMAGE_BACKUP=docker.io/perconalab/percona-server-mong odb-operator:main-backup IMAGE_PMM_CLIENT=docker.io/percona/pmm-client:2.44.1- 1 IMAGE_PMM_SERVER=docker.io/perconalab/pmm-server:dev- latest IMAGE_PMM3_CLIENT=docker.io/perconalab/pmm-client:3-d ev-latest IMAGE_PMM3_SERVER=docker.io/perconalab/pmm-server:3-d ev-latest IMAGE_LOGCOLLECTOR=docker.io/perconalab/fluentbit:mai n-logcollector IMAGE_CLUSTERSYNC=docker.io/percona/percona-clustersy nc-mongodb:0.9.0 CERT_MANAGER_VER=1.19.3 CHAOS_MESH_VER=2.7.1 MINIO_VER=5.4.0 PMM_SERVER_VER=9.9.9 CLEAN_NAMESPACE=1 DELETE_CRD_ON_START=1 SKIP_DELETE=0 SKIP_BACKUPS_TO_AWS_GCP_AZURE= UPDATE_COMPARE_FILES=0 +++ dirname run ++ realpath . + test_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded + . /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../functions +++ dirname /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../functions ++ source /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../vars ++++ mktemp -d +++ export tmp_dir=/tmp/tmp.aI6Yz5fPKz +++ tmp_dir=/tmp/tmp.aI6Yz5fPKz ++++ command -v gsed ++++ command -v sed +++ export sed=/usr/sbin/sed +++ sed=/usr/sbin/sed ++++ command -v gdate ++++ command -v date +++ export date=/usr/sbin/date +++ date=/usr/sbin/date ++++ basename /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded +++ export test_name=demand-backup-sharded +++ test_name=demand-backup-sharded +++ export namespace=demand-backup-sharded-15579 +++ namespace=demand-backup-sharded-15579 +++ export arbiter_namespace=demand-backup-sharded-arbiter-1749 +++ arbiter_namespace=demand-backup-sharded-arbiter-1749 +++ export replica_namespace=demand-backup-sharded-replica-7940 +++ replica_namespace=demand-backup-sharded-replica-7940 ++++ realpath /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../conf +++ export conf_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf +++ conf_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf ++++ realpath /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../.. +++ export src_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058 +++ src_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058 +++ export logs_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../logs +++ logs_dir=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../logs ++++ git rev-parse HEAD +++ export GIT_COMMIT=78121c56762758785b75e9b7c017e15fc887ae5d +++ GIT_COMMIT=78121c56762758785b75e9b7c017e15fc887ae5d +++ export GIT_BRANCH=PR-2058-78121c567 +++ GIT_BRANCH=PR-2058-78121c567 +++ export API=psmdb.percona.com/v1 +++ API=psmdb.percona.com/v1 ++++ awk '{print $2}' ++++++ dirname /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../vars +++++ realpath /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/../../deploy/cr.yaml ++++ grep crVersion /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cr.yaml +++ export OPERATOR_VERSION=1.23.0 +++ OPERATOR_VERSION=1.23.0 +++ export REGISTRY_NAME=docker.io +++ REGISTRY_NAME=docker.io +++ export REGISTRY_NAME_FULL=docker.io/ +++ REGISTRY_NAME_FULL=docker.io/ +++ export IMAGE=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ IMAGE=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ export IMAGE_MONGOD=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ IMAGE_MONGOD=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ export $'IMAGE_MONGOD_CHAIN=docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0' +++ IMAGE_MONGOD_CHAIN=$'docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0' +++ export IMAGE_SEARCH=perconalab/percona-server-mongodb-operator:main-mongot +++ IMAGE_SEARCH=perconalab/percona-server-mongodb-operator:main-mongot +++ export IMAGE_BACKUP=docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ IMAGE_BACKUP=docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ export IMAGE_PMM_CLIENT=docker.io/percona/pmm-client:2.44.1-1 +++ IMAGE_PMM_CLIENT=docker.io/percona/pmm-client:2.44.1-1 +++ export IMAGE_PMM_SERVER=docker.io/perconalab/pmm-server:dev-latest +++ IMAGE_PMM_SERVER=docker.io/perconalab/pmm-server:dev-latest +++ export IMAGE_PMM3_CLIENT=docker.io/perconalab/pmm-client:3-dev-latest +++ IMAGE_PMM3_CLIENT=docker.io/perconalab/pmm-client:3-dev-latest +++ export IMAGE_PMM3_SERVER=docker.io/perconalab/pmm-server:3-dev-latest +++ IMAGE_PMM3_SERVER=docker.io/perconalab/pmm-server:3-dev-latest +++ export IMAGE_LOGCOLLECTOR=docker.io/perconalab/fluentbit:main-logcollector +++ IMAGE_LOGCOLLECTOR=docker.io/perconalab/fluentbit:main-logcollector +++ export IMAGE_CLUSTERSYNC=docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ IMAGE_CLUSTERSYNC=docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE +++ local var=IMAGE +++ local val=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ [[ -z docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 ]] +++ [[ docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 == * * ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ val=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ printf -v IMAGE %s docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_MONGOD +++ local var=IMAGE_MONGOD +++ local val=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ [[ -z docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ]] +++ [[ docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 == * * ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ val=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ printf -v IMAGE_MONGOD %s docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_MONGOD_CHAIN +++ local var=IMAGE_MONGOD_CHAIN +++ local $'val=docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0' +++ [[ -z docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ]] +++ [[ docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 == * * ]] +++ local out= ref +++ IFS= +++ read -r ref +++ [[ -n docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 +++ out+=$'docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\n' +++ IFS= +++ read -r ref +++ [[ -n docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 +++ out+=$'docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\n' +++ IFS= +++ read -r ref +++ [[ -n docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 +++ out+=$'docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0\n' +++ IFS= +++ read -r ref +++ val=$'docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0' +++ printf -v IMAGE_MONGOD_CHAIN %s $'docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0\ndocker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0' +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_BACKUP +++ local var=IMAGE_BACKUP +++ local val=docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ [[ -z docker.io/perconalab/percona-server-mongodb-operator:main-backup ]] +++ [[ docker.io/perconalab/percona-server-mongodb-operator:main-backup == * * ]] ++++ with_registry docker.io/perconalab/percona-server-mongodb-operator:main-backup ++++ local ref=docker.io/perconalab/percona-server-mongodb-operator:main-backup ++++ case "$ref" in ++++ printf %s docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ val=docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ printf -v IMAGE_BACKUP %s docker.io/perconalab/percona-server-mongodb-operator:main-backup +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_PMM_CLIENT +++ local var=IMAGE_PMM_CLIENT +++ local val=docker.io/percona/pmm-client:2.44.1-1 +++ [[ -z docker.io/percona/pmm-client:2.44.1-1 ]] +++ [[ docker.io/percona/pmm-client:2.44.1-1 == * * ]] ++++ with_registry docker.io/percona/pmm-client:2.44.1-1 ++++ local ref=docker.io/percona/pmm-client:2.44.1-1 ++++ case "$ref" in ++++ printf %s docker.io/percona/pmm-client:2.44.1-1 +++ val=docker.io/percona/pmm-client:2.44.1-1 +++ printf -v IMAGE_PMM_CLIENT %s docker.io/percona/pmm-client:2.44.1-1 +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_PMM_SERVER +++ local var=IMAGE_PMM_SERVER +++ local val=docker.io/perconalab/pmm-server:dev-latest +++ [[ -z docker.io/perconalab/pmm-server:dev-latest ]] +++ [[ docker.io/perconalab/pmm-server:dev-latest == * * ]] ++++ with_registry docker.io/perconalab/pmm-server:dev-latest ++++ local ref=docker.io/perconalab/pmm-server:dev-latest ++++ case "$ref" in ++++ printf %s docker.io/perconalab/pmm-server:dev-latest +++ val=docker.io/perconalab/pmm-server:dev-latest +++ printf -v IMAGE_PMM_SERVER %s docker.io/perconalab/pmm-server:dev-latest +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_PMM3_CLIENT +++ local var=IMAGE_PMM3_CLIENT +++ local val=docker.io/perconalab/pmm-client:3-dev-latest +++ [[ -z docker.io/perconalab/pmm-client:3-dev-latest ]] +++ [[ docker.io/perconalab/pmm-client:3-dev-latest == * * ]] ++++ with_registry docker.io/perconalab/pmm-client:3-dev-latest ++++ local ref=docker.io/perconalab/pmm-client:3-dev-latest ++++ case "$ref" in ++++ printf %s docker.io/perconalab/pmm-client:3-dev-latest +++ val=docker.io/perconalab/pmm-client:3-dev-latest +++ printf -v IMAGE_PMM3_CLIENT %s docker.io/perconalab/pmm-client:3-dev-latest +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_PMM3_SERVER +++ local var=IMAGE_PMM3_SERVER +++ local val=docker.io/perconalab/pmm-server:3-dev-latest +++ [[ -z docker.io/perconalab/pmm-server:3-dev-latest ]] +++ [[ docker.io/perconalab/pmm-server:3-dev-latest == * * ]] ++++ with_registry docker.io/perconalab/pmm-server:3-dev-latest ++++ local ref=docker.io/perconalab/pmm-server:3-dev-latest ++++ case "$ref" in ++++ printf %s docker.io/perconalab/pmm-server:3-dev-latest +++ val=docker.io/perconalab/pmm-server:3-dev-latest +++ printf -v IMAGE_PMM3_SERVER %s docker.io/perconalab/pmm-server:3-dev-latest +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_LOGCOLLECTOR +++ local var=IMAGE_LOGCOLLECTOR +++ local val=docker.io/perconalab/fluentbit:main-logcollector +++ [[ -z docker.io/perconalab/fluentbit:main-logcollector ]] +++ [[ docker.io/perconalab/fluentbit:main-logcollector == * * ]] ++++ with_registry docker.io/perconalab/fluentbit:main-logcollector ++++ local ref=docker.io/perconalab/fluentbit:main-logcollector ++++ case "$ref" in ++++ printf %s docker.io/perconalab/fluentbit:main-logcollector +++ val=docker.io/perconalab/fluentbit:main-logcollector +++ printf -v IMAGE_LOGCOLLECTOR %s docker.io/perconalab/fluentbit:main-logcollector +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_CLUSTERSYNC +++ local var=IMAGE_CLUSTERSYNC +++ local val=docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ [[ -z docker.io/percona/percona-clustersync-mongodb:0.9.0 ]] +++ [[ docker.io/percona/percona-clustersync-mongodb:0.9.0 == * * ]] ++++ with_registry docker.io/percona/percona-clustersync-mongodb:0.9.0 ++++ local ref=docker.io/percona/percona-clustersync-mongodb:0.9.0 ++++ case "$ref" in ++++ printf %s docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ val=docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ printf -v IMAGE_CLUSTERSYNC %s docker.io/percona/percona-clustersync-mongodb:0.9.0 +++ for image in IMAGE IMAGE_MONGOD IMAGE_MONGOD_CHAIN IMAGE_BACKUP IMAGE_PMM_CLIENT IMAGE_PMM_SERVER IMAGE_PMM3_CLIENT IMAGE_PMM3_SERVER IMAGE_LOGCOLLECTOR IMAGE_CLUSTERSYNC IMAGE_OPERATOR +++ qualify_image_var IMAGE_OPERATOR +++ local var=IMAGE_OPERATOR +++ local val= +++ [[ -z '' ]] +++ return ++++ kubectl version -o json ++++ jq -r '.serverVersion.major + "." + .serverVersion.minor' ++++ /usr/sbin/sed -r 's/[^0-9.]+//g' Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 +++ export KUBE_VERSION=1.33 +++ KUBE_VERSION=1.33 ++++ detect_k8s_env ++++ local provider=unknown ++++ kubectl api-resources ++++ grep -q openshift.io ++++ kubectl get nodes -o 'jsonpath={.items[0].metadata.labels}' ++++ grep -q eksctl.io ++++ kubectl get nodes -o 'jsonpath={.items[0].metadata.labels}' ++++ grep -q cloud.google.com/gke ++++ provider=gke ++++ echo gke +++ export PLATFORM=gke +++ PLATFORM=gke ++++ echo gke ++++ tr '[:lower:]' '[:upper:]' +++ export GKE=1 +++ GKE=1 +++ export ARCH= +++ ARCH= +++ [[ -z '' ]] ++++ kubectl get nodes -o 'jsonpath={range .items[*]}{.metadata.labels.kubernetes\.io/arch}{"\n"}{end}' ++++ sort -u +++ node_arches=amd64 +++ [[ amd64 == arm64 ]] +++ export CERT_MANAGER_VER=1.19.3 +++ CERT_MANAGER_VER=1.19.3 +++ export MINIO_VER=5.4.0 +++ MINIO_VER=5.4.0 +++ export CHAOS_MESH_VER=2.7.1 +++ CHAOS_MESH_VER=2.7.1 +++ export PMM_SERVER_VER=9.9.9 +++ PMM_SERVER_VER=9.9.9 +++ export SKIP_BACKUPS_TO_AWS_GCP_AZURE=1 +++ SKIP_BACKUPS_TO_AWS_GCP_AZURE=1 +++ export UPDATE_COMPARE_FILES=0 +++ UPDATE_COMPARE_FILES=0 +++ export DELETE_CRD_ON_START=1 +++ DELETE_CRD_ON_START=1 +++ export SKIP_DELETE=0 +++ SKIP_DELETE=0 +++ [[ -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/cloud-secret.yml ]] +++ export SKIP_BACKUPS_TO_AWS_GCP_AZURE= +++ SKIP_BACKUPS_TO_AWS_GCP_AZURE= +++ setup_logging +++ [[ '' == true ]] +++ echo 'Debug logging disabled.' Debug logging disabled. ++ arch_scheduling_enabled ++ [[ -n '' ]] + set_debug + [[ 1 == 1 ]] + set -o xtrace + custom_port=27019 + create_infra demand-backup-sharded-15579 + local ns=demand-backup-sharded-15579 + echo demand-backup-sharded-15579 + [[ 1 == 1 ]] + delete_crd + desc 'get and delete old CRDs and RBAC' + set +o xtrace ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- + kubectl_bin delete -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml --ignore-not-found --wait=false ++ mktemp + local LAST_OUT=/tmp/tmp.2p4XTz5Qj4 ++ mktemp + local LAST_ERR=/tmp/tmp.j55C8BRKTl + local exit_status=0 + local timeout=4 + args=('delete' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml' '--ignore-not-found' '--wait=false') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl delete -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml --ignore-not-found --wait=false + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.2p4XTz5Qj4 + cat /tmp/tmp.j55C8BRKTl + rm /tmp/tmp.2p4XTz5Qj4 /tmp/tmp.j55C8BRKTl + return 0 ++ yq eval .metadata.name /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml ++ grep -v '\-\-\-' grep: warning: stray \ before - grep: warning: stray \ before - + for crd_name in $(yq eval '.metadata.name' "${src_dir}/deploy/crd.yaml" | grep -v '\-\-\-') + kubectl get perconaservermongodbbackups.psmdb.percona.com --all-namespaces -o wide + grep -v NAMESPACE + xargs -L 1 sh -xc 'kubectl patch perconaservermongodbbackups.psmdb.percona.com -n $0 $1 --type=merge -p "{\"metadata\":{\"finalizers\":[]}}"' error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" + : + kubectl_bin wait --for=delete crd perconaservermongodbbackups.psmdb.percona.com ++ mktemp + local LAST_OUT=/tmp/tmp.syepMcycfJ ++ mktemp + local LAST_ERR=/tmp/tmp.sKA6zjGVNC + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'crd' 'perconaservermongodbbackups.psmdb.percona.com') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl wait --for=delete crd perconaservermongodbbackups.psmdb.percona.com + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.syepMcycfJ + cat /tmp/tmp.sKA6zjGVNC + rm /tmp/tmp.syepMcycfJ /tmp/tmp.sKA6zjGVNC + return 0 + for crd_name in $(yq eval '.metadata.name' "${src_dir}/deploy/crd.yaml" | grep -v '\-\-\-') + kubectl get perconaservermongodbclustersyncs.psmdb.percona.com --all-namespaces -o wide + grep -v NAMESPACE + xargs -L 1 sh -xc 'kubectl patch perconaservermongodbclustersyncs.psmdb.percona.com -n $0 $1 --type=merge -p "{\"metadata\":{\"finalizers\":[]}}"' error: the server doesn't have a resource type "perconaservermongodbclustersyncs" + kubectl patch perconaservermongodbclustersyncs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbclustersyncs" + : + kubectl_bin wait --for=delete crd perconaservermongodbclustersyncs.psmdb.percona.com ++ mktemp + local LAST_OUT=/tmp/tmp.uEXGMWtURK ++ mktemp + local LAST_ERR=/tmp/tmp.Od0uZR9Y7O + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'crd' 'perconaservermongodbclustersyncs.psmdb.percona.com') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl wait --for=delete crd perconaservermongodbclustersyncs.psmdb.percona.com + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.uEXGMWtURK + cat /tmp/tmp.Od0uZR9Y7O + rm /tmp/tmp.uEXGMWtURK /tmp/tmp.Od0uZR9Y7O + return 0 + for crd_name in $(yq eval '.metadata.name' "${src_dir}/deploy/crd.yaml" | grep -v '\-\-\-') + grep -v NAMESPACE + kubectl get perconaservermongodbrestores.psmdb.percona.com --all-namespaces -o wide + xargs -L 1 sh -xc 'kubectl patch perconaservermongodbrestores.psmdb.percona.com -n $0 $1 --type=merge -p "{\"metadata\":{\"finalizers\":[]}}"' error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" + : + kubectl_bin wait --for=delete crd perconaservermongodbrestores.psmdb.percona.com ++ mktemp + local LAST_OUT=/tmp/tmp.vBBlqGDum5 ++ mktemp + local LAST_ERR=/tmp/tmp.XK77JKcMuQ + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'crd' 'perconaservermongodbrestores.psmdb.percona.com') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl wait --for=delete crd perconaservermongodbrestores.psmdb.percona.com + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.vBBlqGDum5 + cat /tmp/tmp.XK77JKcMuQ + rm /tmp/tmp.vBBlqGDum5 /tmp/tmp.XK77JKcMuQ + return 0 + for crd_name in $(yq eval '.metadata.name' "${src_dir}/deploy/crd.yaml" | grep -v '\-\-\-') + kubectl get perconaservermongodbs.psmdb.percona.com --all-namespaces -o wide + grep -v NAMESPACE + xargs -L 1 sh -xc 'kubectl patch perconaservermongodbs.psmdb.percona.com -n $0 $1 --type=merge -p "{\"metadata\":{\"finalizers\":[]}}"' error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" + : + kubectl_bin wait --for=delete crd perconaservermongodbs.psmdb.percona.com ++ mktemp + local LAST_OUT=/tmp/tmp.dPpGkep9lU ++ mktemp + local LAST_ERR=/tmp/tmp.1bvlWb7Deb + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'crd' 'perconaservermongodbs.psmdb.percona.com') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl wait --for=delete crd perconaservermongodbs.psmdb.percona.com + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.dPpGkep9lU + cat /tmp/tmp.1bvlWb7Deb + rm /tmp/tmp.dPpGkep9lU /tmp/tmp.1bvlWb7Deb + return 0 + local rbac_yaml=rbac.yaml + '[' -n psmdb-operator ']' + rbac_yaml=cw-rbac.yaml + kubectl_bin delete -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cw-rbac.yaml --ignore-not-found ++ mktemp + local LAST_OUT=/tmp/tmp.kCagmBD9Ls ++ mktemp + local LAST_ERR=/tmp/tmp.kTFUYHe2Fl + local exit_status=0 + local timeout=4 + args=('delete' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cw-rbac.yaml' '--ignore-not-found') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl delete -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cw-rbac.yaml --ignore-not-found + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.kCagmBD9Ls + cat /tmp/tmp.kTFUYHe2Fl + rm /tmp/tmp.kCagmBD9Ls /tmp/tmp.kTFUYHe2Fl + return 0 + check_crd_for_deletion PR-2058-78121c567 + local git_tag=PR-2058-78121c567 ++ curl -s https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/PR-2058-78121c567/deploy/crd.yaml ++ yq eval .metadata.name ++ /usr/sbin/sed s/---//g ++ /usr/sbin/sed ':a;N;$!ba;s/\n/ /g' + for crd_name in $(curl -s https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/${git_tag}/deploy/crd.yaml | yq eval '.metadata.name' | $sed 's/---//g' | $sed ':a;N;$!ba;s/\n/ /g') ++ kubectl_bin get crd/null -o 'jsonpath={.status.conditions[-1].type}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.D0CCbeoM9B +++ mktemp ++ local LAST_ERR=/tmp/tmp.ygzqcxj19x ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'crd/null' '-o' 'jsonpath={.status.conditions[-1].type}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get crd/null -o 'jsonpath={.status.conditions[-1].type}' ++ exit_status=1 ++ set -e ++ '[' 1 '!=' 0 -a -n 1 ']' ++ cat /tmp/tmp.D0CCbeoM9B ++ cat /tmp/tmp.ygzqcxj19x Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ++ sleep 0 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get crd/null -o 'jsonpath={.status.conditions[-1].type}' ++ exit_status=1 ++ set -e ++ '[' 1 '!=' 0 -a -n 1 ']' ++ cat /tmp/tmp.D0CCbeoM9B ++ cat /tmp/tmp.ygzqcxj19x Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ++ sleep 4 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get crd/null -o 'jsonpath={.status.conditions[-1].type}' ++ exit_status=1 ++ set -e ++ '[' 1 '!=' 0 -a -n 1 ']' ++ cat /tmp/tmp.D0CCbeoM9B ++ cat /tmp/tmp.ygzqcxj19x Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ++ sleep 8 ++ cat /tmp/tmp.D0CCbeoM9B ++ cat /tmp/tmp.ygzqcxj19x Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ++ rm /tmp/tmp.D0CCbeoM9B /tmp/tmp.ygzqcxj19x ++ return 1 + [[ '' == Terminating ]] + '[' -n psmdb-operator ']' + create_namespace psmdb-operator + local namespace=psmdb-operator + local skip_clean_namespace= + [[ 1 == 1 ]] + [[ -z '' ]] + destroy_chaos_mesh ++ helm list --all-namespaces --filter chaos-mesh ++ tail -n1 ++ awk '-F ' '{print $2}' ++ sed s/NAMESPACE// + local chaos_mesh_ns= + desc 'destroy chaos-mesh' + set +o xtrace ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- + '[' -n '' ']' ++ kubectl get MutatingWebhookConfiguration ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete MutatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ kubectl get ValidatingWebhookConfiguration ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete ValidatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ kubectl get ValidatingWebhookConfiguration ++ grep validate-auth ++ awk '{print $1}' + timeout 30 kubectl delete ValidatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ kubectl api-resources ++ grep chaos-mesh ++ awk '{print $1}' ++ kubectl get crd ++ grep chaos-mesh.org ++ awk '{print $1}' + timeout 30 kubectl delete crd error: resource(s) were provided, but no name was specified + : ++ kubectl get clusterrolebinding ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete clusterrolebinding error: resource(s) were provided, but no name was specified + : ++ kubectl get clusterrole ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete clusterrole error: resource(s) were provided, but no name was specified + : + desc 'cleaned up all old namespaces' + set +o xtrace ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- + kubectl_bin get ns + grep -E -v '^kube-|^default|Terminating|psmdb-operator|openshift|^gke-|^gmp-|^NAME' + awk '{print$1}' ++ mktemp + [[ -n '' ]] + xargs kubectl delete ns + desc 'cleaned up old namespaces psmdb-operator' + set +o xtrace ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- + kubectl_bin delete namespace psmdb-operator --ignore-not-found ++ mktemp + local LAST_OUT=/tmp/tmp.iiGaIucFmQ ++ mktemp + local LAST_OUT=/tmp/tmp.WgI8nO77AA ++ mktemp + local LAST_ERR=/tmp/tmp.x0uvlDwAM2 + local exit_status=0 + local timeout=4 + args=('get' 'ns') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + local LAST_ERR=/tmp/tmp.TEZNAKvCcy + local exit_status=0 + local timeout=4 + args=('delete' 'namespace' 'psmdb-operator' '--ignore-not-found') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get ns + for i in $(seq 0 2) + set +e + kubectl delete namespace psmdb-operator --ignore-not-found + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.iiGaIucFmQ + cat /tmp/tmp.x0uvlDwAM2 + rm /tmp/tmp.iiGaIucFmQ /tmp/tmp.x0uvlDwAM2 + return 0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.WgI8nO77AA + cat /tmp/tmp.TEZNAKvCcy + rm /tmp/tmp.WgI8nO77AA /tmp/tmp.TEZNAKvCcy + return 0 + kubectl_bin wait --for=delete namespace psmdb-operator ++ mktemp + local LAST_OUT=/tmp/tmp.8xrSDorZwa ++ mktemp + local LAST_ERR=/tmp/tmp.wsKrfoLfbR + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'namespace' 'psmdb-operator') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl wait --for=delete namespace psmdb-operator error: resource(s) were provided, but no name was specified + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.8xrSDorZwa + cat /tmp/tmp.wsKrfoLfbR + rm /tmp/tmp.8xrSDorZwa /tmp/tmp.wsKrfoLfbR + return 0 + desc 'create namespace psmdb-operator' + set +o xtrace ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- + kubectl_bin create namespace psmdb-operator ++ mktemp + local LAST_OUT=/tmp/tmp.xHvYFGDLmP ++ mktemp + local LAST_ERR=/tmp/tmp.zb7y21q9mw + local exit_status=0 + local timeout=4 + args=('create' 'namespace' 'psmdb-operator') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl create namespace psmdb-operator + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.xHvYFGDLmP namespace/psmdb-operator created + cat /tmp/tmp.zb7y21q9mw + rm /tmp/tmp.xHvYFGDLmP /tmp/tmp.zb7y21q9mw + return 0 + set_kube_ctx psmdb-operator + local namespace=psmdb-operator ++ kubectl_bin config current-context +++ mktemp ++ local LAST_OUT=/tmp/tmp.uH0kwCmO2X +++ mktemp ++ local LAST_ERR=/tmp/tmp.iOQg5y7k0f ++ local exit_status=0 ++ local timeout=4 ++ args=('config' 'current-context') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl config current-context ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.uH0kwCmO2X ++ cat /tmp/tmp.iOQg5y7k0f ++ rm /tmp/tmp.uH0kwCmO2X /tmp/tmp.iOQg5y7k0f ++ return 0 + kubectl_bin config set-context gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9 --namespace=psmdb-operator ++ mktemp + local LAST_OUT=/tmp/tmp.4gPG9Cf5X4 ++ mktemp + local LAST_ERR=/tmp/tmp.9Jdt1WQ3HL + local exit_status=0 + local timeout=4 + args=('config' 'set-context' 'gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9' '--namespace=psmdb-operator') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl config set-context gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9 --namespace=psmdb-operator + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.4gPG9Cf5X4 Context "gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9" modified. + cat /tmp/tmp.9Jdt1WQ3HL + rm /tmp/tmp.4gPG9Cf5X4 /tmp/tmp.9Jdt1WQ3HL + return 0 + deploy_operator + desc 'start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567' + set +o xtrace ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 ----------------------------------------------------------------------------------- + local cr_file + '[' -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/crd.yaml ']' + cr_file=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml + kubectl_bin apply --server-side --force-conflicts -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml ++ mktemp + local LAST_OUT=/tmp/tmp.izOaTnmoC8 ++ mktemp + local LAST_ERR=/tmp/tmp.cpQgRYOjIb + local exit_status=0 + local timeout=4 + args=('apply' '--server-side' '--force-conflicts' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply --server-side --force-conflicts -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/crd.yaml + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.izOaTnmoC8 customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbclustersyncs.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied + cat /tmp/tmp.cpQgRYOjIb + rm /tmp/tmp.izOaTnmoC8 /tmp/tmp.cpQgRYOjIb + return 0 + '[' -n psmdb-operator ']' + apply_rbac cw-rbac + local operator_namespace=psmdb-operator + local rbac=cw-rbac + cat /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cw-rbac.yaml + sed -e 's^namespace: .*^namespace: psmdb-operator^' + kubectl_bin apply -n psmdb-operator -f - ++ mktemp + local LAST_OUT=/tmp/tmp.MUEgfuJ40Y ++ mktemp + local LAST_ERR=/tmp/tmp.l6KsXgvDvK + local exit_status=0 + local timeout=4 + args=('apply' '-n' 'psmdb-operator' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -n psmdb-operator -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.MUEgfuJ40Y clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created + cat /tmp/tmp.l6KsXgvDvK + rm /tmp/tmp.MUEgfuJ40Y /tmp/tmp.l6KsXgvDvK + return 0 + add_arch_to_podspecs - + yq eval $'\n\t\t\t(.spec.template.spec.containers[].image = "docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567") |\n\t\t\t((.. | select(.[] == "DISABLE_TELEMETRY")) |= .value="true") |\n\t\t\t((.. | select(.[] == "LOG_LEVEL")) |= .value="DEBUG")' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/deploy/cw-operator.yaml + kubectl_bin apply -n psmdb-operator -f - + arch_scheduling_enabled + [[ -n '' ]] + cat - ++ mktemp + local LAST_OUT=/tmp/tmp.mWalPdhTAN ++ mktemp + local LAST_ERR=/tmp/tmp.vMQ0VSScXu + local exit_status=0 + local timeout=4 + args=('apply' '-n' 'psmdb-operator' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -n psmdb-operator -f - + return + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.mWalPdhTAN deployment.apps/percona-server-mongodb-operator created + cat /tmp/tmp.vMQ0VSScXu + rm /tmp/tmp.mWalPdhTAN /tmp/tmp.vMQ0VSScXu + return 0 + sleep 20 ++ get_operator_pod ++ kubectl_bin get pods --selector=name=percona-server-mongodb-operator -o 'jsonpath={.items[].metadata.name}' -n psmdb-operator +++ mktemp ++ local LAST_OUT=/tmp/tmp.ihhXajjZsh +++ mktemp ++ local LAST_ERR=/tmp/tmp.1ukwVwahiD ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=percona-server-mongodb-operator' '-o' 'jsonpath={.items[].metadata.name}' '-n' 'psmdb-operator') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=percona-server-mongodb-operator -o 'jsonpath={.items[].metadata.name}' -n psmdb-operator ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.ihhXajjZsh ++ cat /tmp/tmp.1ukwVwahiD ++ rm /tmp/tmp.ihhXajjZsh /tmp/tmp.1ukwVwahiD ++ return 0 + wait_operator_pod percona-server-mongodb-operator-66cd7b75cf-f68n2 + local pod=percona-server-mongodb-operator-66cd7b75cf-f68n2 + set +o xtrace waiting for pod/percona-server-mongodb-operator-66cd7b75cf-f68n2 to be ready.OK + echo 'Print operator info from log' Print operator info from log + grep 'Manager starting up' ++ get_operator_pod ++ kubectl_bin get pods --selector=name=percona-server-mongodb-operator -o 'jsonpath={.items[].metadata.name}' -n psmdb-operator +++ mktemp ++ local LAST_OUT=/tmp/tmp.O8NySB5fbA +++ mktemp ++ local LAST_ERR=/tmp/tmp.e5n9Totaeh ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=percona-server-mongodb-operator' '-o' 'jsonpath={.items[].metadata.name}' '-n' 'psmdb-operator') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=percona-server-mongodb-operator -o 'jsonpath={.items[].metadata.name}' -n psmdb-operator ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.O8NySB5fbA ++ cat /tmp/tmp.e5n9Totaeh ++ rm /tmp/tmp.O8NySB5fbA /tmp/tmp.e5n9Totaeh ++ return 0 + kubectl_bin logs -n psmdb-operator percona-server-mongodb-operator-66cd7b75cf-f68n2 ++ mktemp + local LAST_OUT=/tmp/tmp.cDfVHWw4QK ++ mktemp + local LAST_ERR=/tmp/tmp.uWUUopc6Kn + local exit_status=0 + local timeout=4 + args=('logs' '-n' 'psmdb-operator' 'percona-server-mongodb-operator-66cd7b75cf-f68n2') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs -n psmdb-operator percona-server-mongodb-operator-66cd7b75cf-f68n2 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.cDfVHWw4QK + cat /tmp/tmp.uWUUopc6Kn + rm /tmp/tmp.cDfVHWw4QK /tmp/tmp.uWUUopc6Kn + return 0 2026-07-17T00:50:51.500Z INFO setup Manager starting up {"gitCommit": "78121c56762758785b75e9b7c017e15fc887ae5d", "gitBranch": "PR-2058-78121c567", "buildTime": "", "goVersion": "go1.26.5", "os": "linux", "arch": "amd64"} + create_namespace demand-backup-sharded-15579 + local namespace=demand-backup-sharded-15579 + local skip_clean_namespace= + [[ 1 == 1 ]] + [[ -z '' ]] + destroy_chaos_mesh ++ helm list --all-namespaces --filter chaos-mesh ++ tail -n1 ++ awk '-F ' '{print $2}' ++ sed s/NAMESPACE// + local chaos_mesh_ns= + desc 'destroy chaos-mesh' + set +o xtrace ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- + '[' -n '' ']' ++ kubectl get MutatingWebhookConfiguration ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete MutatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ kubectl get ValidatingWebhookConfiguration ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete ValidatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ grep validate-auth ++ kubectl get ValidatingWebhookConfiguration ++ awk '{print $1}' + timeout 30 kubectl delete ValidatingWebhookConfiguration error: resource(s) were provided, but no name was specified + : ++ kubectl api-resources ++ grep chaos-mesh ++ awk '{print $1}' ++ kubectl get crd ++ grep chaos-mesh.org ++ awk '{print $1}' + timeout 30 kubectl delete crd error: resource(s) were provided, but no name was specified + : ++ kubectl get clusterrolebinding ++ awk '{print $1}' ++ grep chaos-mesh + timeout 30 kubectl delete clusterrolebinding error: resource(s) were provided, but no name was specified + : ++ kubectl get clusterrole ++ grep chaos-mesh ++ awk '{print $1}' + timeout 30 kubectl delete clusterrole error: resource(s) were provided, but no name was specified + : + desc 'cleaned up all old namespaces' + set +o xtrace ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- + kubectl_bin get ns + grep -E -v '^kube-|^default|Terminating|psmdb-operator|openshift|^gke-|^gmp-|^NAME' + awk '{print$1}' + [[ -n '' ]] + desc 'cleaned up old namespaces demand-backup-sharded-15579' + xargs kubectl delete ns + set +o xtrace ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-sharded-15579++ mktemp ----------------------------------------------------------------------------------- + kubectl_bin delete namespace demand-backup-sharded-15579 --ignore-not-found ++ mktemp + local LAST_OUT=/tmp/tmp.dDhJx3e1xh ++ mktemp + local LAST_OUT=/tmp/tmp.CGXZOjyysV ++ mktemp + local LAST_ERR=/tmp/tmp.lX7vldkkph + local exit_status=0 + local timeout=4 + args=('get' 'ns') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] + local LAST_ERR=/tmp/tmp.WMwWbyK892 + local exit_status=0 + local timeout=4 + args=('delete' 'namespace' 'demand-backup-sharded-15579' '--ignore-not-found') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled ++ seq 0 2 + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + for i in $(seq 0 2) + set +e + set +e + kubectl get ns + kubectl delete namespace demand-backup-sharded-15579 --ignore-not-found + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.dDhJx3e1xh + cat /tmp/tmp.lX7vldkkph + rm /tmp/tmp.dDhJx3e1xh /tmp/tmp.lX7vldkkph + return 0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.CGXZOjyysV + cat /tmp/tmp.WMwWbyK892 + rm /tmp/tmp.CGXZOjyysV /tmp/tmp.WMwWbyK892 + return 0 + kubectl_bin wait --for=delete namespace demand-backup-sharded-15579 ++ mktemp + local LAST_OUT=/tmp/tmp.u7MPLnXY09 ++ mktemp + local LAST_ERR=/tmp/tmp.YjlaHU0ZUG + local exit_status=0 + local timeout=4 + args=('wait' '--for=delete' 'namespace' 'demand-backup-sharded-15579') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 error: resource(s) were provided, but no name was specified + for i in $(seq 0 2) + set +e + kubectl wait --for=delete namespace demand-backup-sharded-15579 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.u7MPLnXY09 + cat /tmp/tmp.YjlaHU0ZUG + rm /tmp/tmp.u7MPLnXY09 /tmp/tmp.YjlaHU0ZUG + return 0 + desc 'create namespace demand-backup-sharded-15579' + set +o xtrace ----------------------------------------------------------------------------------- create namespace demand-backup-sharded-15579 ----------------------------------------------------------------------------------- + kubectl_bin create namespace demand-backup-sharded-15579 ++ mktemp + local LAST_OUT=/tmp/tmp.1JdcO36F07 ++ mktemp + local LAST_ERR=/tmp/tmp.bYanvX6yGN + local exit_status=0 + local timeout=4 + args=('create' 'namespace' 'demand-backup-sharded-15579') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl create namespace demand-backup-sharded-15579 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.1JdcO36F07 namespace/demand-backup-sharded-15579 created + cat /tmp/tmp.bYanvX6yGN + rm /tmp/tmp.1JdcO36F07 /tmp/tmp.bYanvX6yGN + return 0 + set_kube_ctx demand-backup-sharded-15579 + local namespace=demand-backup-sharded-15579 ++ kubectl_bin config current-context +++ mktemp ++ local LAST_OUT=/tmp/tmp.sE2VnK0nSP +++ mktemp ++ local LAST_ERR=/tmp/tmp.vBU9Yu7Q1x ++ local exit_status=0 ++ local timeout=4 ++ args=('config' 'current-context') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl config current-context ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.sE2VnK0nSP ++ cat /tmp/tmp.vBU9Yu7Q1x ++ rm /tmp/tmp.sE2VnK0nSP /tmp/tmp.vBU9Yu7Q1x ++ return 0 + kubectl_bin config set-context gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9 --namespace=demand-backup-sharded-15579 ++ mktemp + local LAST_OUT=/tmp/tmp.BSyi9CCwBN ++ mktemp + local LAST_ERR=/tmp/tmp.TiQNagQq7F + local exit_status=0 + local timeout=4 + args=('config' 'set-context' 'gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9' '--namespace=demand-backup-sharded-15579') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl config set-context gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9 --namespace=demand-backup-sharded-15579 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.BSyi9CCwBN Context "gke_cloud-dev-112233_us-central1_jen-psmdb-2058-78121c567-22-cluster9" modified. + cat /tmp/tmp.TiQNagQq7F + rm /tmp/tmp.BSyi9CCwBN /tmp/tmp.TiQNagQq7F + return 0 + deploy_minio + local cert_secret= + local service_name=minio-service + desc 'install MinIO: minio-service' + set +o xtrace ----------------------------------------------------------------------------------- install MinIO: minio-service ----------------------------------------------------------------------------------- + helm uninstall minio-service + : + helm repo remove minio "minio" has been removed from your repositories + helm repo add minio https://charts.min.io/ "minio" has been added to your repositories + local endpoint=http://minio-service:9000 + minio_args=('--version' '5.4.0' '--set' 'replicas=1' '--set' 'mode=standalone' '--set' 'resources.requests.memory=256Mi' '--set' 'rootUser=rootuser' '--set' 'rootPassword=rootpass123' '--set' 'users[0].accessKey=some-access-key' '--set' 'users[0].secretKey=some-secret-key' '--set' 'users[0].policy=consoleAdmin' '--set' 'service.type=ClusterIP' '--set' 'configPathmc=/tmp/' '--set' 'securityContext.enabled=false' '--set' 'persistence.size=2G' '--set' 'fullnameOverride=minio-service' '--set' 'serviceAccount.create=true' '--set' 'serviceAccount.name=minio-service-sa') + local minio_args + arch_scheduling_enabled + [[ -n '' ]] + [[ -n '' ]] + retry 10 60 helm install minio-service --version 5.4.0 --set replicas=1 --set mode=standalone --set resources.requests.memory=256Mi --set rootUser=rootuser --set rootPassword=rootpass123 --set 'users[0].accessKey=some-access-key' --set 'users[0].secretKey=some-secret-key' --set 'users[0].policy=consoleAdmin' --set service.type=ClusterIP --set configPathmc=/tmp/ --set securityContext.enabled=false --set persistence.size=2G --set fullnameOverride=minio-service --set serviceAccount.create=true --set serviceAccount.name=minio-service-sa minio/minio + local max=10 + local delay=60 + shift 2 + local n=1 + helm install minio-service --version 5.4.0 --set replicas=1 --set mode=standalone --set resources.requests.memory=256Mi --set rootUser=rootuser --set rootPassword=rootpass123 --set 'users[0].accessKey=some-access-key' --set 'users[0].secretKey=some-secret-key' --set 'users[0].policy=consoleAdmin' --set service.type=ClusterIP --set configPathmc=/tmp/ --set securityContext.enabled=false --set persistence.size=2G --set fullnameOverride=minio-service --set serviceAccount.create=true --set serviceAccount.name=minio-service-sa minio/minio NAME: minio-service LAST DEPLOYED: Fri Jul 17 00:51:30 2026 NAMESPACE: demand-backup-sharded-15579 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.demand-backup-sharded-15579.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace demand-backup-sharded-15579 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace demand-backup-sharded-15579 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace demand-backup-sharded-15579 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace demand-backup-sharded-15579 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local ++ kubectl_bin get pods --selector=release=minio-service -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.zoY3fUTJLn +++ mktemp ++ local LAST_ERR=/tmp/tmp.ty9aqYiwW9 ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=release=minio-service' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=release=minio-service -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.zoY3fUTJLn ++ cat /tmp/tmp.ty9aqYiwW9 ++ rm /tmp/tmp.zoY3fUTJLn /tmp/tmp.ty9aqYiwW9 ++ return 0 + local MINIO_POD=minio-service-6d5f646cdc-8zjbc + wait_pod minio-service-6d5f646cdc-8zjbc + local pod=minio-service-6d5f646cdc-8zjbc + set +o xtrace waiting for pod/minio-service-6d5f646cdc-8zjbc to be ready.OK + '[' -n psmdb-operator ']' + kubectl_bin create svc -n psmdb-operator externalname minio-service --external-name=minio-service.demand-backup-sharded-15579.svc.cluster.local --tcp=9000 service/minio-service created + create_minio_bucket operator-testing http://minio-service:9000 + local bucket=operator-testing + local endpoint=http://minio-service:9000 + aws_cli_minio http://minio-service:9000 s3 mb s3://operator-testing + local endpoint=http://minio-service:9000 + shift + kubectl_bin run -i --rm aws-cli --image=docker.io/amazon/aws-cli:2.34.60 --restart=Never --env=AWS_ACCESS_KEY_ID=some-access-key --env=AWS_SECRET_ACCESS_KEY=some-secret-key --env=AWS_DEFAULT_REGION=us-east-1 -- --no-verify-ssl --endpoint-url http://minio-service:9000 s3 mb s3://operator-testing ++ mktemp + local LAST_OUT=/tmp/tmp.7AFcG84LSL ++ mktemp + local LAST_ERR=/tmp/tmp.YAiJTiLZyZ + local exit_status=0 + local timeout=4 + args=('run' '-i' '--rm' 'aws-cli' '--image=docker.io/amazon/aws-cli:2.34.60' '--restart=Never' '--env=AWS_ACCESS_KEY_ID=some-access-key' '--env=AWS_SECRET_ACCESS_KEY=some-secret-key' '--env=AWS_DEFAULT_REGION=us-east-1' '--' '--no-verify-ssl' '--endpoint-url' 'http://minio-service:9000' 's3' 'mb' 's3://operator-testing') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl run -i --rm aws-cli --image=docker.io/amazon/aws-cli:2.34.60 --restart=Never --env=AWS_ACCESS_KEY_ID=some-access-key --env=AWS_SECRET_ACCESS_KEY=some-secret-key --env=AWS_DEFAULT_REGION=us-east-1 -- --no-verify-ssl --endpoint-url http://minio-service:9000 s3 mb s3://operator-testing + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.7AFcG84LSL make_bucket: operator-testing make_bucket: operator-testing pod "aws-cli" deleted from demand-backup-sharded-15579 namespace + cat /tmp/tmp.YAiJTiLZyZ All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: Internal error occurred: unable to upgrade connection: container aws-cli not found in pod aws-cli_demand-backup-sharded-15579 + rm /tmp/tmp.7AFcG84LSL /tmp/tmp.YAiJTiLZyZ + return 0 + desc 'create first PSMDB cluster with custom port 27019' + set +o xtrace ----------------------------------------------------------------------------------- create first PSMDB cluster with custom port 27019 ----------------------------------------------------------------------------------- + cluster=some-name + kubectl_bin apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/secrets.yml -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/client.yml ++ mktemp + local LAST_OUT=/tmp/tmp.s6RGk3zM2S ++ mktemp + local LAST_ERR=/tmp/tmp.mcFlaDkXnM + local exit_status=0 + local timeout=4 + args=('apply' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/secrets.yml' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/client.yml') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/secrets.yml -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/client.yml + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.s6RGk3zM2S secret/some-users created deployment.apps/psmdb-client created + cat /tmp/tmp.mcFlaDkXnM + rm /tmp/tmp.s6RGk3zM2S /tmp/tmp.mcFlaDkXnM + return 0 + apply_s3_storage_secrets + desc 'create secrets for cloud storages' + set +o xtrace ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- + '[' -z '' ']' + kubectl_bin apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/minio-secret.yml -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/cloud-secret.yml ++ mktemp + local LAST_OUT=/tmp/tmp.Jia9CMS2dc ++ mktemp + local LAST_ERR=/tmp/tmp.oQqUa5DmLv + local exit_status=0 + local timeout=4 + args=('apply' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/minio-secret.yml' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/cloud-secret.yml') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/minio-secret.yml -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/cloud-secret.yml + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.Jia9CMS2dc secret/minio-secret created secret/aws-s3-secret created secret/alibaba-cloud-secret created secret/oci-secret created secret/gcp-cs-secret created secret/azure-secret created secret/gcp-cs-sa-key-secret created + cat /tmp/tmp.oQqUa5DmLv + rm /tmp/tmp.Jia9CMS2dc /tmp/tmp.oQqUa5DmLv + return 0 + create_runtime_class + kubectl_bin delete runtimeclass container-rc --ignore-not-found ++ mktemp + local LAST_OUT=/tmp/tmp.iE3LdZID4A ++ mktemp + local LAST_ERR=/tmp/tmp.MhjH4eGAnu + local exit_status=0 + local timeout=4 + args=('delete' 'runtimeclass' 'container-rc' '--ignore-not-found') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl delete runtimeclass container-rc --ignore-not-found + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.iE3LdZID4A runtimeclass.node.k8s.io "container-rc" deleted + cat /tmp/tmp.MhjH4eGAnu + rm /tmp/tmp.iE3LdZID4A /tmp/tmp.MhjH4eGAnu + return 0 + local handler=runc + desc 'Create RuntimeClass with handler runc' + set +o xtrace ----------------------------------------------------------------------------------- Create RuntimeClass with handler runc ----------------------------------------------------------------------------------- + yq eval '.handler = "runc"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/conf/container-rc.yaml + kubectl_bin apply -f - ++ mktemp + local LAST_OUT=/tmp/tmp.w36zAYWiDe ++ mktemp + local LAST_ERR=/tmp/tmp.urt2hs6O6D + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.w36zAYWiDe runtimeclass.node.k8s.io/container-rc created + cat /tmp/tmp.urt2hs6O6D + rm /tmp/tmp.w36zAYWiDe /tmp/tmp.urt2hs6O6D + return 0 + apply_cluster /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/some-name-rs0.yml + '[' -z '' ']' + cat_config /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/some-name-rs0.yml + kubectl_bin apply -f - + cat /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/some-name-rs0.yml + yq eval '(.spec | select(.image == null)).image = "docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0"' + yq eval '(.spec | select(has("pmm"))).pmm.image = "docker.io/percona/pmm-client:2.44.1-1"' ++ mktemp + yq eval '(.spec | select(has("initImage"))).initImage = "docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567"' + yq eval '(.spec | select(has("backup"))).backup.image = "docker.io/perconalab/percona-server-mongodb-operator:main-backup"' + local LAST_OUT=/tmp/tmp.T7nXnA66RM ++ mktemp + [[ -n docker.io/perconalab/fluentbit:main-logcollector ]] + IMAGE_LOGCOLLECTOR=docker.io/perconalab/fluentbit:main-logcollector + yq eval '(.spec | select(has("logcollector"))).logcollector.image = strenv(IMAGE_LOGCOLLECTOR)' + yq eval '(.spec | select(has("search"))).search.image = "perconalab/percona-server-mongodb-operator:main-mongot"' + local LAST_ERR=/tmp/tmp.FYIpNO1bVu + yq eval '.spec.upgradeOptions.apply="Never"' + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + add_arch_to_psmdb - + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] + /usr/sbin/sed -e s/NAME_SPACE/demand-backup-sharded-15579/g + arch_scheduling_enabled + [[ -n '' ]] + cat - ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + return + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.T7nXnA66RM perconaservermongodb.psmdb.percona.com/some-name created + cat /tmp/tmp.FYIpNO1bVu + rm /tmp/tmp.T7nXnA66RM /tmp/tmp.FYIpNO1bVu + return 0 + desc 'check if all 3 Pods started' + set +o xtrace ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- + wait_for_running some-name-rs0 3 + local name=some-name-rs0 + let last_pod=2 + local check_cluster_readiness=true + set_debug + [[ 1 == 1 ]] + set -o xtrace + local rs_name=rs0 + local cluster_name=some-name ++ seq 0 2 + for i in $(seq 0 $last_pod) + [[ 0 -eq 2 ]] + wait_pod some-name-rs0-0 + local pod=some-name-rs0-0 + set +o xtrace waiting for pod/some-name-rs0-0 to be ready.....OK + for i in $(seq 0 $last_pod) + [[ 1 -eq 2 ]] + wait_pod some-name-rs0-1 + local pod=some-name-rs0-1 + set +o xtrace waiting for pod/some-name-rs0-1 to be ready.......OK + for i in $(seq 0 $last_pod) + [[ 2 -eq 2 ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].arbiter.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.HdK1zdAfR1 +++ mktemp ++ local LAST_ERR=/tmp/tmp.ShMTGmdCv2 ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="rs0")].arbiter.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].arbiter.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.HdK1zdAfR1 ++ cat /tmp/tmp.ShMTGmdCv2 ++ rm /tmp/tmp.HdK1zdAfR1 /tmp/tmp.ShMTGmdCv2 ++ return 0 + [[ '' == true ]] + wait_pod some-name-rs0-2 + local pod=some-name-rs0-2 + set +o xtrace waiting for pod/some-name-rs0-2 to be ready.....OK ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].nonvoting.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.VCBJqg54R5 +++ mktemp ++ local LAST_ERR=/tmp/tmp.Gf3xjcGKPl ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="rs0")].nonvoting.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].nonvoting.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.VCBJqg54R5 ++ cat /tmp/tmp.Gf3xjcGKPl ++ rm /tmp/tmp.VCBJqg54R5 /tmp/tmp.Gf3xjcGKPl ++ return 0 + [[ '' == true ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].hidden.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.50JEHqRMUv +++ mktemp ++ local LAST_ERR=/tmp/tmp.VD5C8AT90K ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="rs0")].hidden.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="rs0")].hidden.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.50JEHqRMUv ++ cat /tmp/tmp.VD5C8AT90K ++ rm /tmp/tmp.50JEHqRMUv /tmp/tmp.VD5C8AT90K ++ return 0 + [[ '' == true ]] + sleep 10 + [[ true == true ]] + set +x Waiting for cluster some-name to be ready............................................ + wait_for_running some-name-cfg 3 false + local name=some-name-cfg + let last_pod=2 + local check_cluster_readiness=false + set_debug + [[ 1 == 1 ]] + set -o xtrace + local rs_name=cfg + local cluster_name=some-name ++ seq 0 2 + for i in $(seq 0 $last_pod) + [[ 0 -eq 2 ]] + wait_pod some-name-cfg-0 + local pod=some-name-cfg-0 + set +o xtrace waiting for pod/some-name-cfg-0 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 1 -eq 2 ]] + wait_pod some-name-cfg-1 + local pod=some-name-cfg-1 + set +o xtrace waiting for pod/some-name-cfg-1 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 2 -eq 2 ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].arbiter.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.V9hW69SN5K +++ mktemp ++ local LAST_ERR=/tmp/tmp.gp45WgiNvZ ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="cfg")].arbiter.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].arbiter.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.V9hW69SN5K ++ cat /tmp/tmp.gp45WgiNvZ ++ rm /tmp/tmp.V9hW69SN5K /tmp/tmp.gp45WgiNvZ ++ return 0 + [[ '' == true ]] + wait_pod some-name-cfg-2 + local pod=some-name-cfg-2 + set +o xtrace waiting for pod/some-name-cfg-2 to be ready.OK ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].nonvoting.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.OXPTM8m4q9 +++ mktemp ++ local LAST_ERR=/tmp/tmp.edoW0xqPbY ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="cfg")].nonvoting.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].nonvoting.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.OXPTM8m4q9 ++ cat /tmp/tmp.edoW0xqPbY ++ rm /tmp/tmp.OXPTM8m4q9 /tmp/tmp.edoW0xqPbY ++ return 0 + [[ '' == true ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].hidden.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.mEvzp9CJ8Y +++ mktemp ++ local LAST_ERR=/tmp/tmp.1ArpFpQpXh ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="cfg")].hidden.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="cfg")].hidden.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.mEvzp9CJ8Y ++ cat /tmp/tmp.1ArpFpQpXh ++ rm /tmp/tmp.mEvzp9CJ8Y /tmp/tmp.1ArpFpQpXh ++ return 0 + [[ '' == true ]] + sleep 10 + [[ false == true ]] + wait_for_running some-name-mongos 3 + local name=some-name-mongos + let last_pod=2 + local check_cluster_readiness=true + set_debug + [[ 1 == 1 ]] + set -o xtrace + local rs_name=mongos + local cluster_name=some-name ++ seq 0 2 + for i in $(seq 0 $last_pod) + [[ 0 -eq 2 ]] + wait_pod some-name-mongos-0 + local pod=some-name-mongos-0 + set +o xtrace waiting for pod/some-name-mongos-0 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 1 -eq 2 ]] + wait_pod some-name-mongos-1 + local pod=some-name-mongos-1 + set +o xtrace waiting for pod/some-name-mongos-1 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 2 -eq 2 ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.fV7t2ND2Te +++ mktemp ++ local LAST_ERR=/tmp/tmp.tCb2Cvbjx4 ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.fV7t2ND2Te ++ cat /tmp/tmp.tCb2Cvbjx4 ++ rm /tmp/tmp.fV7t2ND2Te /tmp/tmp.tCb2Cvbjx4 ++ return 0 + [[ '' == true ]] + wait_pod some-name-mongos-2 + local pod=some-name-mongos-2 + set +o xtrace waiting for pod/some-name-mongos-2 to be ready.OK ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.BEhy3Dk4fu +++ mktemp ++ local LAST_ERR=/tmp/tmp.OinAYHhpYD ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.BEhy3Dk4fu ++ cat /tmp/tmp.OinAYHhpYD ++ rm /tmp/tmp.BEhy3Dk4fu /tmp/tmp.OinAYHhpYD ++ return 0 + [[ '' == true ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.WBGXyLB193 +++ mktemp ++ local LAST_ERR=/tmp/tmp.SV461wPY2o ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.WBGXyLB193 ++ cat /tmp/tmp.SV461wPY2o ++ rm /tmp/tmp.WBGXyLB193 /tmp/tmp.SV461wPY2o ++ return 0 + [[ '' == true ]] + sleep 10 + [[ true == true ]] + set +x Waiting for cluster some-name to be ready + sleep 10 + desc 'check if service and statefulset created with expected config' + set +o xtrace ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- + compare_kubectl statefulset/some-name-rs0 + local resource=statefulset/some-name-rs0 + local postfix= + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs0.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs0.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-rs0 + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - ++ mktemp + local LAST_OUT=/tmp/tmp.FnDd4Z8irX ++ mktemp + local LAST_ERR=/tmp/tmp.DiuO4dK666 + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-rs0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-rs0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.FnDd4Z8irX + cat /tmp/tmp.DiuO4dK666 + rm /tmp/tmp.FnDd4Z8irX /tmp/tmp.DiuO4dK666 + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs0.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs0.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs0.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs0.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs0.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs0.yml + log 'compare_kubectl: statefulset/some-name-rs0 OK' + set +o xtrace [2026-07-17T00:56:04+0000] compare_kubectl: statefulset/some-name-rs0 OK + compare_kubectl statefulset/some-name-rs1 + local resource=statefulset/some-name-rs1 + local postfix= + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs1.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs1.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-rs1 + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - ++ mktemp + local LAST_OUT=/tmp/tmp.vWwjWz1T72 ++ mktemp + local LAST_ERR=/tmp/tmp.5mShqi1pp5 + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-rs1') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-rs1 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.vWwjWz1T72 + cat /tmp/tmp.5mShqi1pp5 + rm /tmp/tmp.vWwjWz1T72 /tmp/tmp.5mShqi1pp5 + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs1.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs1.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs1.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs1.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs1.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs1.yml + log 'compare_kubectl: statefulset/some-name-rs1 OK' + set +o xtrace [2026-07-17T00:56:05+0000] compare_kubectl: statefulset/some-name-rs1 OK + compare_kubectl statefulset/some-name-rs2 + local resource=statefulset/some-name-rs2 + local postfix= + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs2.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs2.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-rs2 + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - ++ mktemp + local LAST_OUT=/tmp/tmp.BFXKSz7OCD ++ mktemp + local LAST_ERR=/tmp/tmp.pbaRaNasxx + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-rs2') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-rs2 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.BFXKSz7OCD + cat /tmp/tmp.pbaRaNasxx + rm /tmp/tmp.BFXKSz7OCD /tmp/tmp.pbaRaNasxx + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs2.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs2.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs2.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs2.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-rs2.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-rs2.yml + log 'compare_kubectl: statefulset/some-name-rs2 OK' + set +o xtrace [2026-07-17T00:56:06+0000] compare_kubectl: statefulset/some-name-rs2 OK + compare_kubectl statefulset/some-name-cfg + local resource=statefulset/some-name-cfg + local postfix= + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-cfg.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-cfg.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-cfg ++ mktemp + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - + local LAST_OUT=/tmp/tmp.DAm9ri9jh6 ++ mktemp + local LAST_ERR=/tmp/tmp.GQ2WoKoTwt + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-cfg') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-cfg + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.DAm9ri9jh6 + cat /tmp/tmp.GQ2WoKoTwt + rm /tmp/tmp.DAm9ri9jh6 /tmp/tmp.GQ2WoKoTwt + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-cfg.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-cfg.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-cfg.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-cfg.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-cfg.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-cfg.yml + log 'compare_kubectl: statefulset/some-name-cfg OK' + set +o xtrace [2026-07-17T00:56:07+0000] compare_kubectl: statefulset/some-name-cfg OK + compare_kubectl statefulset/some-name-mongos '' + local resource=statefulset/some-name-mongos + local postfix= + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-mongos + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - ++ mktemp + local LAST_OUT=/tmp/tmp.Ta8m5QgtyO ++ mktemp + local LAST_ERR=/tmp/tmp.LJItfZja9K + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-mongos') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-mongos + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.Ta8m5QgtyO + cat /tmp/tmp.LJItfZja9K + rm /tmp/tmp.Ta8m5QgtyO /tmp/tmp.LJItfZja9K + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + log 'compare_kubectl: statefulset/some-name-mongos OK' + set +o xtrace [2026-07-17T00:56:08+0000] compare_kubectl: statefulset/some-name-mongos OK + kubectl_bin apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/mongos-secret.yml ++ mktemp + local LAST_OUT=/tmp/tmp.F04GoEDXi7 ++ mktemp + local LAST_ERR=/tmp/tmp.aJAzDrBVfh + local exit_status=0 + local timeout=4 + args=('apply' '-f' '/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/mongos-secret.yml') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/mongos-secret.yml + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.F04GoEDXi7 secret/some-name-mongos created + cat /tmp/tmp.aJAzDrBVfh + rm /tmp/tmp.F04GoEDXi7 /tmp/tmp.aJAzDrBVfh + return 0 + sleep 10 + wait_for_running some-name-mongos 3 + local name=some-name-mongos + let last_pod=2 + local check_cluster_readiness=true + set_debug + [[ 1 == 1 ]] + set -o xtrace + local rs_name=mongos + local cluster_name=some-name ++ seq 0 2 + for i in $(seq 0 $last_pod) + [[ 0 -eq 2 ]] + wait_pod some-name-mongos-0 + local pod=some-name-mongos-0 + set +o xtrace waiting for pod/some-name-mongos-0 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 1 -eq 2 ]] + wait_pod some-name-mongos-1 + local pod=some-name-mongos-1 + set +o xtrace waiting for pod/some-name-mongos-1 to be ready.OK + for i in $(seq 0 $last_pod) + [[ 2 -eq 2 ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.0fLcAj8TJf +++ mktemp ++ local LAST_ERR=/tmp/tmp.7Y7QRAaV6S ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].arbiter.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.0fLcAj8TJf ++ cat /tmp/tmp.7Y7QRAaV6S ++ rm /tmp/tmp.0fLcAj8TJf /tmp/tmp.7Y7QRAaV6S ++ return 0 + [[ '' == true ]] + wait_pod some-name-mongos-2 + local pod=some-name-mongos-2 + set +o xtrace waiting for pod/some-name-mongos-2 to be ready.OK ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.t9R2ObuoZj +++ mktemp ++ local LAST_ERR=/tmp/tmp.AqW3CMmi5B ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].nonvoting.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.t9R2ObuoZj ++ cat /tmp/tmp.AqW3CMmi5B ++ rm /tmp/tmp.t9R2ObuoZj /tmp/tmp.AqW3CMmi5B ++ return 0 + [[ '' == true ]] ++ kubectl_bin get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.zdRHIRkqJs +++ mktemp ++ local LAST_ERR=/tmp/tmp.dy4V3Gdswb ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'psmdb' 'some-name' '-o' 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get psmdb some-name -o 'jsonpath={.spec.replsets[?(@.name=="mongos")].hidden.enabled}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.zdRHIRkqJs ++ cat /tmp/tmp.dy4V3Gdswb ++ rm /tmp/tmp.zdRHIRkqJs /tmp/tmp.dy4V3Gdswb ++ return 0 + [[ '' == true ]] + sleep 10 + [[ true == true ]] + set +x Waiting for cluster some-name to be ready.................................................................................................. + compare_kubectl statefulset/some-name-mongos -secret + local resource=statefulset/some-name-mongos + local postfix=-secret + local skip_generation_check= + local expected_result=/mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos-secret.yml + local new_result=/tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + [[ -n '' ]] + kubectl_bin get -o yaml statefulset/some-name-mongos + yq eval $'\n\t\t\tdel(.metadata.ownerReferences[].apiVersion) |\n\t\t\tdel(.metadata.managedFields) |\n\t\t\tdel(.. | select(has("creationTimestamp")).creationTimestamp) |\n\t\t\tdel(.. | select(has("namespace")).namespace) |\n\t\t\tdel(.. | select(has("uid")).uid) |\n\t\t\tdel(.metadata.resourceVersion) |\n\t\t\tdel(.spec.template.spec.containers[].env[] | select(.name == "NAMESPACE")) |\n\t\t\tdel(.metadata.selfLink) |\n\t\t\tdel(.metadata.annotations."cloud.google.com/neg") |\n\t\t\tdel(.metadata.annotations."field.cattle.io/publicEndpoints") |\n\t\t\tdel(.metadata.annotations."metallb.io/ip-allocated-from-pool") |\n\t\t\tdel(.metadata.annotations."networking.gke.io/target-pool") |\n\t\t\tdel(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |\n\t\t\tdel(.. | select(has("image")).image) |\n\t\t\tdel(.. | select(has("clusterIP")).clusterIP) |\n\t\t\tdel(.. | select(has("clusterIPs")).clusterIPs) |\n\t\t\tdel(.. | select(has("dataSource")).dataSource) |\n\t\t\tdel(.. | select(has("procMount")).procMount) |\n\t\t\tdel(.. | select(has("storageClassName")).storageClassName) |\n\t\t\tdel(.. | select(has("finalizers")).finalizers) |\n\t\t\tdel(.. | select(has("kubernetes.io/pvc-protection"))."kubernetes.io/pvc-protection") |\n\t\t\tdel(.. | select(has("volumeName")).volumeName) |\n\t\t\tdel(.. | select(has("volume.beta.kubernetes.io/storage-provisioner"))."volume.beta.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/storage-provisioner"))."volume.kubernetes.io/storage-provisioner") |\n\t\t\tdel(.spec.volumeMode) |\n\t\t\tdel(.. | select(has("volume.kubernetes.io/selected-node"))."volume.kubernetes.io/selected-node") |\n\t\t\tdel(.. | select(has("percona.com/last-config-hash"))."percona.com/last-config-hash") |\n\t\t\tdel(.. | select(has("percona.com/configuration-hash"))."percona.com/configuration-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-hash"))."percona.com/ssl-hash") |\n\t\t\tdel(.. | select(has("percona.com/ssl-internal-hash"))."percona.com/ssl-internal-hash") |\n\t\t\tdel(.spec.volumeClaimTemplates[].spec.volumeMode | select(. == "Filesystem")) |\n\t\t\tdel(.. | select(has("healthCheckNodePort")).healthCheckNodePort) |\n\t\t\tdel(.. | select(has("nodePort")).nodePort) |\n\t\t\tdel(.status) |\n\t\t\t(.. | select(tag == "!!str")) |= sub("demand-backup-sharded-15579", "NAME_SPACE") |\n\t\t\tdel(.spec.volumeClaimTemplates[].apiVersion) |\n\t\t\tdel(.spec.volumeClaimTemplates[].kind) |\n\t\t\tdel(.spec.ipFamilies) |\n\t\t\tdel(.spec.ipFamilyPolicy) |\n\t\t\t(.. | select(. == "extensions/v1beta1")) = "apps/v1" |\n\t\t\t(.. | select(. == "batch/v1beta1")) = "batch/v1" ' - ++ mktemp + local LAST_OUT=/tmp/tmp.qxO9q7H7c7 ++ mktemp + local LAST_ERR=/tmp/tmp.6jk5EFw09C + local exit_status=0 + local timeout=4 + args=('get' '-o' 'yaml' 'statefulset/some-name-mongos') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl get -o yaml statefulset/some-name-mongos + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.qxO9q7H7c7 + cat /tmp/tmp.6jk5EFw09C + rm /tmp/tmp.qxO9q7H7c7 /tmp/tmp.6jk5EFw09C + return 0 + yq -i eval 'del(.spec.persistentVolumeClaimRetentionPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + arch_scheduling_enabled + [[ -n '' ]] + version_gt 1.22 ++ echo '1.33 >= 1.22' ++ bc -l + '[' 1 -eq 1 ']' + return 0 + yq -i eval 'del(.spec.internalTrafficPolicy)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + yq -i eval 'del(.spec.allocateLoadBalancerNodePorts)' /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + [[ /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos-secret.yml == */cronjob* ]] + '[' -n '' ']' + [[ 0 -eq 0 ]] + diff -u /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/statefulset_some-name-mongos-secret.yml /tmp/tmp.aI6Yz5fPKz/statefulset_some-name-mongos.yml + log 'compare_kubectl: statefulset/some-name-mongos OK' + set +o xtrace [2026-07-17T01:00:09+0000] compare_kubectl: statefulset/some-name-mongos OK + desc 'checking hookscripts' + set +o xtrace ----------------------------------------------------------------------------------- checking hookscripts ----------------------------------------------------------------------------------- + kubectl_bin logs some-name-rs0-0 backup-agent + grep -q 'Backup Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.hIMiMfrqqL ++ mktemp + local LAST_ERR=/tmp/tmp.taIi6saj4b + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs0-0' 'backup-agent') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs0-0 backup-agent + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.hIMiMfrqqL + cat /tmp/tmp.taIi6saj4b + rm /tmp/tmp.hIMiMfrqqL /tmp/tmp.taIi6saj4b + return 0 + kubectl_bin logs some-name-rs0-0 + grep -q 'RS0 Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.Zu2IWopCE2 ++ mktemp + local LAST_ERR=/tmp/tmp.ovYS4jNWFd + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs0-0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs0-0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.Zu2IWopCE2 + cat /tmp/tmp.ovYS4jNWFd Defaulted container "mongod" out of: mongod, backup-agent, mongo-init (init) + rm /tmp/tmp.Zu2IWopCE2 /tmp/tmp.ovYS4jNWFd + return 0 + kubectl_bin logs some-name-rs1-0 backup-agent + grep -q 'Backup Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.m11SNDVScs ++ mktemp + local LAST_ERR=/tmp/tmp.QClvmJ3GbG + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs1-0' 'backup-agent') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs1-0 backup-agent + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.m11SNDVScs + cat /tmp/tmp.QClvmJ3GbG + rm /tmp/tmp.m11SNDVScs /tmp/tmp.QClvmJ3GbG + return 0 + kubectl_bin logs some-name-rs1-0 + grep -q 'RS1 Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.sApYmoJznz ++ mktemp + local LAST_ERR=/tmp/tmp.x1mZ4q1xWK + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs1-0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs1-0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.sApYmoJznz + cat /tmp/tmp.x1mZ4q1xWK Defaulted container "mongod" out of: mongod, backup-agent, mongo-init (init) + rm /tmp/tmp.sApYmoJznz /tmp/tmp.x1mZ4q1xWK + return 0 + kubectl_bin logs some-name-rs2-0 backup-agent + grep -q 'Backup Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.4Dqs55EEvp ++ mktemp + local LAST_ERR=/tmp/tmp.4Joa0n2hNk + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs2-0' 'backup-agent') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs2-0 backup-agent + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.4Dqs55EEvp + cat /tmp/tmp.4Joa0n2hNk + rm /tmp/tmp.4Dqs55EEvp /tmp/tmp.4Joa0n2hNk + return 0 + kubectl_bin logs some-name-rs2-0 + grep -q 'RS2 Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.uZthlznKXm ++ mktemp + local LAST_ERR=/tmp/tmp.lCavHX41aq + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-rs2-0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-rs2-0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.uZthlznKXm + cat /tmp/tmp.lCavHX41aq Defaulted container "mongod" out of: mongod, rs-sidecar-1, backup-agent, mongo-init (init) + rm /tmp/tmp.uZthlznKXm /tmp/tmp.lCavHX41aq + return 0 + kubectl_bin logs some-name-cfg-0 backup-agent + grep -q 'Backup Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.edUrh9Fuff ++ mktemp + local LAST_ERR=/tmp/tmp.Q5PXbOpJ9E + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-cfg-0' 'backup-agent') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-cfg-0 backup-agent + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.edUrh9Fuff + cat /tmp/tmp.Q5PXbOpJ9E + rm /tmp/tmp.edUrh9Fuff /tmp/tmp.Q5PXbOpJ9E + return 0 + kubectl_bin logs some-name-cfg-0 ++ mktemp + grep -q 'CFG Hookscript started' + local LAST_OUT=/tmp/tmp.KWnqiXsgEh ++ mktemp + local LAST_ERR=/tmp/tmp.nKgQDgm6Ap + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-cfg-0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-cfg-0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.KWnqiXsgEh + cat /tmp/tmp.nKgQDgm6Ap Defaulted container "mongod" out of: mongod, cfg-sidecar-1, backup-agent, mongo-init (init) + rm /tmp/tmp.KWnqiXsgEh /tmp/tmp.nKgQDgm6Ap + return 0 + kubectl_bin logs some-name-mongos-0 + grep -q 'Mongos Hookscript started' ++ mktemp + local LAST_OUT=/tmp/tmp.iP6Ey3sixN ++ mktemp + local LAST_ERR=/tmp/tmp.ptBytbnSX0 + local exit_status=0 + local timeout=4 + args=('logs' 'some-name-mongos-0') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl logs some-name-mongos-0 + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.iP6Ey3sixN + cat /tmp/tmp.ptBytbnSX0 Defaulted container "mongos" out of: mongos, mongos-sidecar-1, mongo-init (init) + rm /tmp/tmp.iP6Ey3sixN /tmp/tmp.ptBytbnSX0 + return 0 + desc 'write data, read from all' + set +o xtrace ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- + run_mongos 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"},{db:"myApp1",role:"readWrite"},{db:"myApp2",role:"readWrite"}]})' userAdmin:userAdmin123456@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"},{db:"myApp1",role:"readWrite"},{db:"myApp2",role:"readWrite"}]})' + local uri=userAdmin:userAdmin123456@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.YUOHeOJ0Po +++ mktemp ++ local LAST_ERR=/tmp/tmp.tv7TxvUq5m ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.YUOHeOJ0Po ++ cat /tmp/tmp.tv7TxvUq5m ++ rm /tmp/tmp.YUOHeOJ0Po /tmp/tmp.tv7TxvUq5m ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"},{db:"myApp1",role:"readWrite"},{db:"myApp2",role:"readWrite"}]})\n'\'' | mongo mongodb://userAdmin:userAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.QZtLN2fAuJ ++ mktemp + local LAST_ERR=/tmp/tmp.W3uq8vFk01 + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"},{db:"myApp1",role:"readWrite"},{db:"myApp2",role:"readWrite"}]})\n'\'' | mongo mongodb://userAdmin:userAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"},{db:"myApp1",role:"readWrite"},{db:"myApp2",role:"readWrite"}]})\n'\'' | mongo mongodb://userAdmin:userAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.QZtLN2fAuJ Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("f3d1b0dd-716f-431b-b9ee-ccaf29f6dfc6") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" }, { "db" : "myApp1", "role" : "readWrite" }, { "db" : "myApp2", "role" : "readWrite" } ] } bye + cat /tmp/tmp.W3uq8vFk01 + rm /tmp/tmp.QZtLN2fAuJ /tmp/tmp.W3uq8vFk01 + return 0 + run_mongos 'sh.enableSharding("myApp","rs0")' clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=sh.enableSharding("myApp","rs0")' + local uri=clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ awk -F: '{print $2}' ++ echo .svc.cluster.local + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.zxfFmHbqBW +++ mktemp ++ local LAST_ERR=/tmp/tmp.zmMWWLALEp ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.zxfFmHbqBW ++ cat /tmp/tmp.zmMWWLALEp ++ rm /tmp/tmp.zxfFmHbqBW /tmp/tmp.zmMWWLALEp ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp","rs0")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.IabCuC6J8c ++ mktemp + local LAST_ERR=/tmp/tmp.GakkitnwQD + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''sh.enableSharding("myApp","rs0")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp","rs0")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.IabCuC6J8c Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("56b9399c-332e-4b6b-a57b-ebd834db2da9") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1784250029, 8), "signature" : { "hash" : BinData(0,"rKz9U7kzW4tw4XDTsjKqH28F8ew="), "keyId" : NumberLong("7663294023498465300") } }, "operationTime" : Timestamp(1784250029, 5) } bye + cat /tmp/tmp.GakkitnwQD + rm /tmp/tmp.IabCuC6J8c /tmp/tmp.GakkitnwQD + return 0 + run_mongos 'sh.enableSharding("myApp1","rs1")' clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=sh.enableSharding("myApp1","rs1")' + local uri=clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.zJCIYGXMQi +++ mktemp ++ local LAST_ERR=/tmp/tmp.Hwjhq3kDpp ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.zJCIYGXMQi ++ cat /tmp/tmp.Hwjhq3kDpp ++ rm /tmp/tmp.zJCIYGXMQi /tmp/tmp.Hwjhq3kDpp ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp1","rs1")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.UVk5qJHPvM ++ mktemp + local LAST_ERR=/tmp/tmp.4DBGANERkQ + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''sh.enableSharding("myApp1","rs1")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp1","rs1")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.UVk5qJHPvM Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("207d2cb1-58ba-4474-8359-f4129beb22a5") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1784250031, 8), "signature" : { "hash" : BinData(0,"qi1gHX2z90R1EHxEVPlX3h+L+O8="), "keyId" : NumberLong("7663294023498465300") } }, "operationTime" : Timestamp(1784250031, 5) } bye + cat /tmp/tmp.4DBGANERkQ + rm /tmp/tmp.UVk5qJHPvM /tmp/tmp.4DBGANERkQ + return 0 + run_mongos 'sh.enableSharding("myApp2","rs2")' clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=sh.enableSharding("myApp2","rs2")' + local uri=clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.SPCasIujPz +++ mktemp ++ local LAST_ERR=/tmp/tmp.6UBd3jYzRo ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.SPCasIujPz ++ cat /tmp/tmp.6UBd3jYzRo ++ rm /tmp/tmp.SPCasIujPz /tmp/tmp.6UBd3jYzRo ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp2","rs2")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.37U1wC6gN0 ++ mktemp + local LAST_ERR=/tmp/tmp.RAW8IRYGer + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''sh.enableSharding("myApp2","rs2")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''sh.enableSharding("myApp2","rs2")\n'\'' | mongo mongodb://clusterAdmin:clusterAdmin123456@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.37U1wC6gN0 Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("9d93ae67-cf6c-4237-a213-11d74ca23dc3") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1784250034, 8), "signature" : { "hash" : BinData(0,"3dbfsbsrSsr4RrDrGVc31NtFXCY="), "keyId" : NumberLong("7663294023498465300") } }, "operationTime" : Timestamp(1784250034, 5) } bye + cat /tmp/tmp.RAW8IRYGer + rm /tmp/tmp.37U1wC6gN0 /tmp/tmp.RAW8IRYGer + return 0 + insert_data_mongos 100500 myApp '' 27019 + local data=100500 + local db_name=myApp + local flags= + local port=27019 + run_mongos 'use myApp\n db.test.insert({ x: 100500 })' myApp:myPass@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=use myApp\n db.test.insert({ x: 100500 })' + local uri=myApp:myPass@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ awk -F: '{print $2}' ++ echo .svc.cluster.local + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.dLF6W0wk1L +++ mktemp ++ local LAST_ERR=/tmp/tmp.tRfsPJZjiG ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.dLF6W0wk1L ++ cat /tmp/tmp.tRfsPJZjiG ++ rm /tmp/tmp.dLF6W0wk1L /tmp/tmp.tRfsPJZjiG ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.EqEjQwSBJx ++ mktemp + local LAST_ERR=/tmp/tmp.teDm2PGHGG + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''use myApp\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.EqEjQwSBJx Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("09623b00-cee9-410f-ac61-227ff7f04414") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye + cat /tmp/tmp.teDm2PGHGG + rm /tmp/tmp.EqEjQwSBJx /tmp/tmp.teDm2PGHGG + return 0 + insert_data_mongos 100500 myApp1 '' 27019 + local data=100500 + local db_name=myApp1 + local flags= + local port=27019 + run_mongos 'use myApp1\n db.test.insert({ x: 100500 })' myApp:myPass@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=use myApp1\n db.test.insert({ x: 100500 })' + local uri=myApp:myPass@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.RoZ3Oc2lLW +++ mktemp ++ local LAST_ERR=/tmp/tmp.91KkvstGYG ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.RoZ3Oc2lLW ++ cat /tmp/tmp.91KkvstGYG ++ rm /tmp/tmp.RoZ3Oc2lLW /tmp/tmp.91KkvstGYG ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp1\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.I5uuF5G9SF ++ mktemp + local LAST_ERR=/tmp/tmp.4omOR181Ld + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''use myApp1\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp1\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.I5uuF5G9SF Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("ad9ecf19-6dc4-4852-8cf2-c0729fa10049") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match switched to db myApp1 WriteResult({ "nInserted" : 1 }) bye + cat /tmp/tmp.4omOR181Ld + rm /tmp/tmp.I5uuF5G9SF /tmp/tmp.4omOR181Ld + return 0 + insert_data_mongos 100500 myApp2 '' 27019 + local data=100500 + local db_name=myApp2 + local flags= + local port=27019 + run_mongos 'use myApp2\n db.test.insert({ x: 100500 })' myApp:myPass@some-name-mongos.demand-backup-sharded-15579 '' '' '' 27019 + local 'command=use myApp2\n db.test.insert({ x: 100500 })' + local uri=myApp:myPass@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.gFwfIDnWXw +++ mktemp ++ local LAST_ERR=/tmp/tmp.H1BlXdGZor ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.gFwfIDnWXw ++ cat /tmp/tmp.H1BlXdGZor ++ rm /tmp/tmp.gFwfIDnWXw /tmp/tmp.H1BlXdGZor ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp2\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.MTza8Gg7Gm ++ mktemp + local LAST_ERR=/tmp/tmp.Vpx4TNvoAu + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''use myApp2\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp2\n db.test.insert({ x: 100500 })\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.MTza8Gg7Gm Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("916e5131-c285-469f-a8ec-b8777f9b257e") } Percona Server for MongoDB server version: v8.0.26-11 WARNING: shell and server versions do not match switched to db myApp2 WriteResult({ "nInserted" : 1 }) bye + cat /tmp/tmp.Vpx4TNvoAu + rm /tmp/tmp.MTza8Gg7Gm /tmp/tmp.Vpx4TNvoAu + return 0 + minikube_sleep + sleep_time=10 + [[ '' == 1 ]] + compare_mongos_cmd find myApp:myPass@some-name-mongos.demand-backup-sharded-15579 '' '' '' '' 27019 + local command=find + local uri=myApp:myPass@some-name-mongos.demand-backup-sharded-15579 + local postfix= + local suffix= + local database=myApp + local collection=test + local port=27019 + local tls=false + [[ false == true ]] + mongos_command=run_mongos + log 'running db.test.command() in myApp' + set +o xtrace [2026-07-17T01:00:42+0000] running db.test.command() in myApp + run_mongos 'use myApp\n db.test.find()' myApp:myPass@some-name-mongos.demand-backup-sharded-15579 mongodb '' '' 27019 + local 'command=use myApp\n db.test.find()' + local uri=myApp:myPass@some-name-mongos.demand-backup-sharded-15579 + local driver=mongodb + local suffix=.svc.cluster.local + local mongo_flag= + local port=27019 + local mongo_bin=mongo + grep -E -v 'I NETWORK|W NETWORK|Error saving history file|Percona Server for MongoDB|connecting to:|Unable to reach primary for set|Implicit session:|versions do not match|Error saving history file:|Started a new thread for the timer service' + /usr/sbin/sed -re 's/ObjectId\("[0-9a-f]+"\)//; s/-[0-9]+.svc/-xxx.svc/' ++ echo .svc.cluster.local ++ awk -F: '{print $2}' + suffix_port= + [[ -z '' ]] + suffix=.svc.cluster.local:27019 ++ kubectl_bin get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' +++ mktemp ++ local LAST_OUT=/tmp/tmp.Nj9v9siHQA +++ mktemp ++ local LAST_ERR=/tmp/tmp.0WIOtxi37K ++ local exit_status=0 ++ local timeout=4 ++ args=('get' 'pods' '--selector=name=psmdb-client' '-o' 'jsonpath={.items[].metadata.name}') ++ local args ++ transformed_files=() ++ local transformed_files ++ arch_scheduling_enabled ++ [[ -n '' ]] ++ arch_scheduling_enabled ++ [[ -n '' ]] +++ seq 0 2 ++ for i in $(seq 0 2) ++ set +e ++ kubectl get pods --selector=name=psmdb-client -o 'jsonpath={.items[].metadata.name}' ++ exit_status=0 ++ set -e ++ '[' 0 '!=' 0 -a -n 1 ']' ++ break ++ cat /tmp/tmp.Nj9v9siHQA ++ cat /tmp/tmp.0WIOtxi37K ++ rm /tmp/tmp.Nj9v9siHQA /tmp/tmp.0WIOtxi37K ++ return 0 + local client_container=psmdb-client-757cd65695-qb45g + kubectl_bin exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' ++ mktemp + local LAST_OUT=/tmp/tmp.CGRbQHP9WR ++ mktemp + local LAST_ERR=/tmp/tmp.SnCaqI0YUN + local exit_status=0 + local timeout=4 + args=('exec' 'psmdb-client-757cd65695-qb45g' '--' 'bash' '-c' 'export HOME=/tmp; printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl exec psmdb-client-757cd65695-qb45g -- bash -c 'export HOME=/tmp; printf '\''use myApp\n db.test.find()\n'\'' | mongo mongodb://myApp:myPass@some-name-mongos.demand-backup-sharded-15579.svc.cluster.local:27019/admin ' + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.CGRbQHP9WR + cat /tmp/tmp.SnCaqI0YUN + rm /tmp/tmp.CGRbQHP9WR /tmp/tmp.SnCaqI0YUN + return 0 + [[ 0 -eq 0 ]] + diff /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/compare/find.json /tmp/tmp.aI6Yz5fPKz/find + wait_backup_agent some-name-rs0-0 + local agent_pod=some-name-rs0-0 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs0-0...2026-07-17T00:55:12.000+0000 I listening for the commands + wait_backup_agent some-name-rs0-1 + local agent_pod=some-name-rs0-1 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs0-1...2026-07-17T00:55:10.000+0000 I listening for the commands + wait_backup_agent some-name-rs0-2 + local agent_pod=some-name-rs0-2 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs0-2...2026-07-17T00:55:17.000+0000 I listening for the commands + wait_backup_agent some-name-rs1-0 + local agent_pod=some-name-rs1-0 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs1-0...2026-07-17T00:55:14.000+0000 I listening for the commands + wait_backup_agent some-name-rs1-1 + local agent_pod=some-name-rs1-1 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs1-1...2026-07-17T00:55:16.000+0000 I listening for the commands + wait_backup_agent some-name-rs1-2 + local agent_pod=some-name-rs1-2 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs1-2...2026-07-17T00:55:21.000+0000 I listening for the commands + wait_backup_agent some-name-rs2-0 + local agent_pod=some-name-rs2-0 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs2-0...2026-07-17T00:55:14.000+0000 I listening for the commands + wait_backup_agent some-name-rs2-1 + local agent_pod=some-name-rs2-1 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs2-1...2026-07-17T00:55:17.000+0000 I listening for the commands + wait_backup_agent some-name-rs2-2 + local agent_pod=some-name-rs2-2 + set +o xtrace waiting for pbm-agent to be ready in some-name-rs2-2...2026-07-17T00:55:17.000+0000 I listening for the commands + backup_name_aws=backup-aws-s3 + backup_name_minio=backup-minio + backup_name_gcp=backup-gcp-cs + backup_name_azure=backup-azure-blob + backup_name_oss_native=backup-oss-native + desc 'run backups' + set +o xtrace ----------------------------------------------------------------------------------- run backups ----------------------------------------------------------------------------------- + run_backup minio + local storage=minio + local backup_name=backup-minio + local type=logical + log 'running backup backup-minio' + set +o xtrace [2026-07-17T01:00:57+0000] running backup backup-minio + kubectl_bin apply -f - + yq eval $'.metadata.name = "backup-minio"\n\t\t\t| .spec.storageName = "minio"\n\t\t\t| .spec.type = "logical"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/backup-minio.yml ++ mktemp + local LAST_OUT=/tmp/tmp.Bcip8lrk2d ++ mktemp + local LAST_ERR=/tmp/tmp.nUDa72o6OG + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.Bcip8lrk2d perconaservermongodbbackup.psmdb.percona.com/backup-minio created + cat /tmp/tmp.nUDa72o6OG + rm /tmp/tmp.Bcip8lrk2d /tmp/tmp.nUDa72o6OG + return 0 + '[' -z '' ']' + run_backup aws-s3 + local storage=aws-s3 + local backup_name=backup-aws-s3 + local type=logical + log 'running backup backup-aws-s3' + set +o xtrace [2026-07-17T01:00:59+0000] running backup backup-aws-s3 + kubectl_bin apply -f - + yq eval $'.metadata.name = "backup-aws-s3"\n\t\t\t| .spec.storageName = "aws-s3"\n\t\t\t| .spec.type = "logical"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/backup-aws-s3.yml ++ mktemp + local LAST_OUT=/tmp/tmp.0y9hLUDvEn ++ mktemp + local LAST_ERR=/tmp/tmp.5wTJSXubYe + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.0y9hLUDvEn perconaservermongodbbackup.psmdb.percona.com/backup-aws-s3 created + cat /tmp/tmp.5wTJSXubYe + rm /tmp/tmp.0y9hLUDvEn /tmp/tmp.5wTJSXubYe + return 0 + run_backup gcp-cs + local storage=gcp-cs + local backup_name=backup-gcp-cs + local type=logical + log 'running backup backup-gcp-cs' + set +o xtrace [2026-07-17T01:01:01+0000] running backup backup-gcp-cs + yq eval $'.metadata.name = "backup-gcp-cs"\n\t\t\t| .spec.storageName = "gcp-cs"\n\t\t\t| .spec.type = "logical"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/backup-gcp-cs.yml + kubectl_bin apply -f - ++ mktemp + local LAST_OUT=/tmp/tmp.pMmiEDQA6c ++ mktemp + local LAST_ERR=/tmp/tmp.dJ80aJbZvT + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.pMmiEDQA6c perconaservermongodbbackup.psmdb.percona.com/backup-gcp-cs created + cat /tmp/tmp.dJ80aJbZvT + rm /tmp/tmp.pMmiEDQA6c /tmp/tmp.dJ80aJbZvT + return 0 + run_backup azure-blob + local storage=azure-blob + local backup_name=backup-azure-blob + local type=logical + log 'running backup backup-azure-blob' + set +o xtrace [2026-07-17T01:01:03+0000] running backup backup-azure-blob + kubectl_bin apply -f - + yq eval $'.metadata.name = "backup-azure-blob"\n\t\t\t| .spec.storageName = "azure-blob"\n\t\t\t| .spec.type = "logical"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/backup-azure-blob.yml ++ mktemp + local LAST_OUT=/tmp/tmp.lQFgjUrB9b ++ mktemp + local LAST_ERR=/tmp/tmp.scMEgZMHma + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.lQFgjUrB9b perconaservermongodbbackup.psmdb.percona.com/backup-azure-blob created + cat /tmp/tmp.scMEgZMHma + rm /tmp/tmp.lQFgjUrB9b /tmp/tmp.scMEgZMHma + return 0 + run_backup oss-native + local storage=oss-native + local backup_name=backup-oss-native + local type=logical + log 'running backup backup-oss-native' + set +o xtrace [2026-07-17T01:01:06+0000] running backup backup-oss-native + yq eval $'.metadata.name = "backup-oss-native"\n\t\t\t| .spec.storageName = "oss-native"\n\t\t\t| .spec.type = "logical"' /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/demand-backup-sharded/conf/backup-oss-native.yml + kubectl_bin apply -f - ++ mktemp + local LAST_OUT=/tmp/tmp.kxXjpKM0bt ++ mktemp + local LAST_ERR=/tmp/tmp.LrH1oj2Dfx + local exit_status=0 + local timeout=4 + args=('apply' '-f' '-') + local args + transformed_files=() + local transformed_files + arch_scheduling_enabled + [[ -n '' ]] + arch_scheduling_enabled + [[ -n '' ]] ++ seq 0 2 + for i in $(seq 0 2) + set +e + kubectl apply -f - + exit_status=0 + set -e + '[' 0 '!=' 0 -a -n 1 ']' + break + cat /tmp/tmp.kxXjpKM0bt perconaservermongodbbackup.psmdb.percona.com/backup-oss-native created + cat /tmp/tmp.LrH1oj2Dfx + rm /tmp/tmp.kxXjpKM0bt /tmp/tmp.LrH1oj2Dfx + return 0 + wait_backup backup-aws-s3 + local backup_name=backup-aws-s3 + local target_state=ready + set +o xtrace waiting for backup-aws-s3 to reach ready state........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................2026-07-17T01:20:54.751Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "df0c5b27-97bc-4976-a126-3abb840e27a8", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:20:54.835Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "df0c5b27-97bc-4976-a126-3abb840e27a8", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:54.835Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "df0c5b27-97bc-4976-a126-3abb840e27a8", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:54.836Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "df0c5b27-97bc-4976-a126-3abb840e27a8"} 2026-07-17T01:20:55.036Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0"} 2026-07-17T01:20:55.052Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0", "state": "waiting"} 2026-07-17T01:20:55.138Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:20:55.138Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:20:55.312Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:55.312Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:55.314Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "e647024f-472e-4ac6-b8c5-8226c8cb3ed0"} 2026-07-17T01:20:55.446Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94"} 2026-07-17T01:20:55.461Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94", "state": "running"} 2026-07-17T01:20:55.548Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:20:55.548Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:20:55.637Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:55.639Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:20:55.642Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "ae7eae37-4c0a-4f1d-b8c1-35b9805b0d94"} 2026-07-17T01:20:55.657Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b"} 2026-07-17T01:20:55.674Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b", "state": "waiting"} 2026-07-17T01:20:55.762Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:20:55.762Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:20:55.848Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:55.848Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:55.850Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "85f8d97e-967a-4f00-bce6-610331624a0b"} 2026-07-17T01:20:59.534Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049"} 2026-07-17T01:20:59.551Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049", "state": "waiting"} 2026-07-17T01:20:59.636Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:20:59.636Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049", "currentJob": {"Name":"backup-aws-s3","Type":0}} 2026-07-17T01:20:59.721Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:59.721Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:20:59.722Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "036b9948-66b5-40e1-8feb-29e61f860049"} 2026-07-17T01:20:59.838Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146"} 2026-07-17T01:20:59.855Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146", "state": "waiting"} 2026-07-17T01:20:59.954Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:20:59.954Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:21:00.055Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:00.055Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:00.056Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "1c81ce49-fa97-4ac3-ac44-311c25988146"} 2026-07-17T01:21:00.315Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0"} 2026-07-17T01:21:00.328Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0", "state": "waiting"} 2026-07-17T01:21:00.415Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:00.415Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:21:00.503Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:00.503Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:00.504Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "77b33824-2cf3-4de4-b179-b59ff8fc77b0"} 2026-07-17T01:21:00.643Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b"} 2026-07-17T01:21:00.666Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b", "state": "running"} 2026-07-17T01:21:00.756Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:00.756Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:21:00.847Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:00.848Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:21:00.851Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "56e0425b-48f0-4f5e-82d4-76429d9a351b"} 2026-07-17T01:21:00.851Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd"} 2026-07-17T01:21:00.872Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd", "state": "waiting"} 2026-07-17T01:21:00.952Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:00.952Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:21:01.041Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:01.041Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:01.043Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "fd10b670-b7f4-4173-a729-05b2cb0919dd"} 2026-07-17T01:21:04.723Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6"} 2026-07-17T01:21:04.736Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6", "state": "waiting"} 2026-07-17T01:21:04.828Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:04.828Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6", "currentJob": {"Name":"backup-aws-s3","Type":0}} 2026-07-17T01:21:04.918Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:04.918Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:04.920Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "463af10c-e6a5-4783-9446-4f825e8c62b6"} 2026-07-17T01:21:05.057Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e"} 2026-07-17T01:21:05.068Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e", "state": "waiting"} 2026-07-17T01:21:05.158Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:05.158Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:21:05.243Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:05.243Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:05.244Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "994e4dfb-1977-44bf-9383-507fcce33b9e"} 2026-07-17T01:21:05.505Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1"} 2026-07-17T01:21:05.519Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1", "state": "waiting"} 2026-07-17T01:21:05.608Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:05.608Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:21:05.692Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:05.692Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:05.693Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "63e754bc-8c79-4333-839f-5e7ba00329e1"} 2026-07-17T01:21:05.852Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c"} 2026-07-17T01:21:05.867Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c", "state": "running"} 2026-07-17T01:21:05.951Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:05.951Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:21:06.042Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:06.044Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:21:06.047Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "544d206d-e679-4fb2-a9d0-dbb806116f7c"} 2026-07-17T01:21:06.047Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1"} 2026-07-17T01:21:06.071Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1", "state": "waiting"} 2026-07-17T01:21:06.162Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:06.162Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:21:06.246Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:06.246Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:06.248Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "04442563-9f05-4bf2-8c6f-8f8dc89bccb1"} 2026-07-17T01:21:09.921Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d"} 2026-07-17T01:21:09.943Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d", "state": "waiting"} 2026-07-17T01:21:10.037Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:10.037Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d", "currentJob": {"Name":"backup-aws-s3","Type":0}} 2026-07-17T01:21:10.128Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.128Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.130Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "a1a7d9a5-1832-4c37-9bb7-86129887835d"} 2026-07-17T01:21:10.245Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47"} 2026-07-17T01:21:10.260Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47", "state": "waiting"} 2026-07-17T01:21:10.349Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:10.349Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:21:10.440Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.440Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.442Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "dd33d2ef-cb63-4ad0-a654-efa3d8b54b47"} 2026-07-17T01:21:10.695Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e"} 2026-07-17T01:21:10.712Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e", "state": "waiting"} 2026-07-17T01:21:10.802Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:10.802Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:21:10.891Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.892Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:10.893Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "27c53bf0-8b0b-4bdd-89c3-bfa09e70b16e"} 2026-07-17T01:21:11.047Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693"} 2026-07-17T01:21:11.061Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693", "state": "running"} 2026-07-17T01:21:11.145Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:11.145Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:21:11.234Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:11.235Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:21:11.238Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "5691917c-c73f-40c9-912f-131fc863f693"} 2026-07-17T01:21:11.248Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3"} 2026-07-17T01:21:11.272Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3", "state": "waiting"} 2026-07-17T01:21:11.357Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:11.357Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:21:11.442Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:11.442Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:11.443Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "f1836481-ca8b-4966-8021-aad7af0fadc3"} 2026-07-17T01:21:15.131Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b"} 2026-07-17T01:21:15.166Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b", "state": "waiting"} 2026-07-17T01:21:15.271Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:15.271Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b", "currentJob": {"Name":"backup-aws-s3","Type":0}} 2026-07-17T01:21:15.382Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:15.382Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:15.384Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "53350b0a-105a-4fbd-b401-ce338b6e7b0b"} 2026-07-17T01:21:15.442Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d"} 2026-07-17T01:21:15.495Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d", "state": "waiting"} 2026-07-17T01:21:15.600Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:15.600Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:21:15.701Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:15.702Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:15.703Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "10dc9a5c-0cfa-4624-8e82-25915db7d78d"} 2026-07-17T01:21:15.894Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a"} 2026-07-17T01:21:15.907Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a", "state": "waiting"} 2026-07-17T01:21:15.995Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:15.995Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:21:16.083Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:16.083Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:16.084Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "563bc94c-2423-4767-9d27-0233f65b3a9a"} 2026-07-17T01:21:16.239Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30"} 2026-07-17T01:21:16.345Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30", "state": "running"} 2026-07-17T01:21:16.438Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:16.438Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:21:16.526Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:16.528Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:21:16.531Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "e2b372ae-6eee-497a-bee1-46dd01d90a30"} 2026-07-17T01:21:16.531Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0"} 2026-07-17T01:21:16.545Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0", "state": "waiting"} 2026-07-17T01:21:16.630Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:16.630Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:21:16.720Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:16.720Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:16.721Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "341639d2-32cf-45cb-87ad-c6a0dfd94dd0"} 2026-07-17T01:21:20.385Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec"} 2026-07-17T01:21:20.397Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec", "state": "waiting"} 2026-07-17T01:21:20.483Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:20.483Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec", "currentJob": {"Name":"backup-aws-s3","Type":0}} 2026-07-17T01:21:20.569Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:20.569Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:20.571Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-aws-s3","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-aws-s3", "reconcileID": "ccbb6a33-31e9-4d39-8e45-cbac5b6620ec"} 2026-07-17T01:21:20.704Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf"} 2026-07-17T01:21:20.717Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf", "state": "waiting"} 2026-07-17T01:21:20.807Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:20.807Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf", "currentJob": {"Name":"backup-oss-native","Type":0}} 2026-07-17T01:21:20.901Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:20.901Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:20.902Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-oss-native","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-oss-native", "reconcileID": "774dd4c8-ae8a-4056-a853-a1547e3078bf"} 2026-07-17T01:21:21.085Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f"} 2026-07-17T01:21:21.103Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f", "state": "waiting"} 2026-07-17T01:21:21.193Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:21.193Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f", "currentJob": {"Name":"backup-gcp-cs","Type":0}} 2026-07-17T01:21:21.294Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:21.294Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:21.296Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-gcp-cs","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-gcp-cs", "reconcileID": "bf42320f-68b5-4a99-a787-847f9248a33f"} 2026-07-17T01:21:21.533Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b"} 2026-07-17T01:21:21.547Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b", "state": "running"} 2026-07-17T01:21:21.634Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:21.634Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b", "currentJob": {"Name":"backup-minio","Type":0}} 2026-07-17T01:21:21.722Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:21.723Z DEBUG backupStatus Got backup meta {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b", "backup": "backup-minio", "pbmName": "2026-07-17T01:00:59Z", "meta": {"type":"logical","opid":"6a597ecb82164ca5124fe378","name":"2026-07-17T01:00:59Z","shardRemap":{"cfg":"config"},"replsets":[{"name":"rs1","backup_name":"2026-07-17T01:00:59Z/rs1/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs1/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250057,"I":3},"last_write_ts":{"T":1784250063,"I":61},"node":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"},{"timestamp":1784250077,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs0","backup_name":"2026-07-17T01:00:59Z/rs0/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs0/oplog","start_ts":1784250060,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250056,"I":1},"last_write_ts":{"T":1784250064,"I":9},"node":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250076,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"cfg","backup_name":"2026-07-17T01:00:59Z/cfg/metadata.json","oplog_name":"2026-07-17T01:00:59Z/cfg/oplog","start_ts":1784250060,"status":"dumpDone","size":0,"size_uncompressed":0,"iscs":true,"last_transition_ts":1784250063,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1784250064,"I":45},"node":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","conditions":[{"timestamp":1784250060,"status":"running"},{"timestamp":1784250063,"status":"dumpDone"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"},{"name":"rs2","backup_name":"2026-07-17T01:00:59Z/rs2/metadata.json","oplog_name":"2026-07-17T01:00:59Z/rs2/oplog","start_ts":1784250061,"status":"error","size":0,"size_uncompressed":0,"last_transition_ts":1784250106,"first_write_ts":{"T":1784250059,"I":1},"last_write_ts":{"T":1784250064,"I":28},"node":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","error":"waiting for done: backup stuck, last beat ts: 1784250075","conditions":[{"timestamp":1784250061,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"},{"timestamp":1784250079,"status":"done"},{"timestamp":1784250106,"status":"error","error":"waiting for done: backup stuck, last beat ts: 1784250075"}],"pbm_version":"2.15.0","mongo_version":"8.0.26-11"}],"compression":"gzip","store":{"name":"minio","profile":true,"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","credentials":{"access-key-id":"***","secret-access-key":"***","vault":{}},"maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":false}},"size":35830,"size_uncompressed":0,"mongodb_version":"8.0.26-11","fcv":"8.0","start_ts":1784250059,"last_transition_ts":1784250064,"first_write_ts":{"T":1784250060,"I":26},"last_write_ts":{"T":1,"I":1},"hb":{"T":1784250075,"I":5},"status":"dumpDone","conditions":[{"timestamp":1784250059,"status":"starting"},{"timestamp":1784250063,"status":"running"},{"timestamp":1784250064,"status":"dumpDone"}],"n":[{"rs":"cfg","n":["some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018","some-name-cfg-2.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"],"ack":"some-name-cfg-1.some-name-cfg.demand-backup-sharded-15579.svc.cluster.local:27018"},{"rs":"rs0","n":["some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs0-2.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs0-1.some-name-rs0.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs2","n":["some-name-rs2-1.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs2-2.some-name-rs2.demand-backup-sharded-15579.svc.cluster.local:27016"},{"rs":"rs1","n":["some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016","some-name-rs1-2.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"],"ack":"some-name-rs1-1.some-name-rs1.demand-backup-sharded-15579.svc.cluster.local:27016"}],"pbm_version":"2.15.0","balancer":"full"}} 2026-07-17T01:21:21.726Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-minio", "reconcileID": "8fb31c85-f221-4c0c-8930-8c559453c32b"} 2026-07-17T01:21:21.726Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e"} 2026-07-17T01:21:21.740Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e", "state": "waiting"} 2026-07-17T01:21:21.842Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e", "cluster": "some-name", "namespace": "demand-backup-sharded-15579"} 2026-07-17T01:21:21.842Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e", "currentJob": {"Name":"backup-azure-blob","Type":0}} 2026-07-17T01:21:21.943Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:21.943Z INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e", "lease": "psmdb-some-name-backup-lock"} 2026-07-17T01:21:21.945Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-azure-blob","namespace":"demand-backup-sharded-15579"}, "namespace": "demand-backup-sharded-15579", "name": "backup-azure-blob", "reconcileID": "1af3e6e9-bdfa-4ed1-ae68-78b3c5ea7c8e"} NAME CLUSTER STORAGE DESTINATION TYPE SIZE STATUS COMPLETED AGE backup-aws-s3 some-name aws-s3 waiting 20m backup-azure-blob some-name azure-blob waiting 20m backup-gcp-cs some-name gcp-cs waiting 20m backup-minio some-name minio s3://operator-testing/2026-07-17T01:00:59Z logical running 20m backup-oss-native some-name oss-native waiting 20m Name: backup-aws-s3 Namespace: demand-backup-sharded-15579 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBBackup Metadata: Creation Timestamp: 2026-07-17T01:01:01Z Finalizers: percona.com/delete-backup Generation: 1 Resource Version: 1784250062088591018 UID: b4c46aa9-4f4e-4174-aa9d-a20b9867ac58 Spec: Cluster Name: some-name Starting Deadline Seconds: 120 Storage Name: aws-s3 Type: logical Status: State: waiting Events: Backup object psmdb-backup/backup-aws-s3 is in waiting state. [01:21:30.785064] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. [01:21:31.802423] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. [01:21:33.825483] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. [01:21:34.300240] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. [01:21:35.262614] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. [01:21:36.383431] WARNING kubectl error: No resources found in pitr-to-new-cluster-32363 namespace. FAILED =================================== FAILURES =================================== ______________________________ test_bash_wrapper _______________________________ request = > def test_bash_wrapper(request: pytest.FixtureRequest) -> None: """Run a bash test script via pytest wrapper.""" test_name = request.config.getoption("--test-name") if not test_name: pytest.skip("No --test-name provided") > run_bash_test(test_name) e2e-tests/test_pytest_wrapper.py:10: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_name = 'demand-backup-sharded' def run_bash_test(test_name: str) -> None: """Run bash script with stdout/stderr inherited from the pytest process.""" script_path = Path(__file__).parent.parent / test_name / "run" if not script_path.exists(): pytest.fail(f"Script not found: {script_path}") original_cwd = os.getcwd() script_dir = script_path.parent try: os.chdir(script_dir) result = subprocess.run(["bash", "run"]) if result.returncode != 0: > pytest.fail(f"Test {test_name} failed with exit code {result.returncode}") E Failed: Test demand-backup-sharded failed with exit code 1 e2e-tests/lib/bash_wrapper.py:27: Failed ------------------------------ Captured log setup ------------------------------ WARNING lib.kubectl:kubectl.py:18 kubectl error: Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 INFO conftest:conftest.py:182 Environment variables: KUBE_VERSION=1.33 EKS=0 GKE=1 OPENSHIFT= MINIKUBE= API=psmdb.percona.com/v1 REGISTRY_NAME=docker.io GIT_COMMIT=78121c56762758785b75e9b7c017e15fc887ae5d GIT_BRANCH=PR-2058 OPERATOR_VERSION=1.23.0 IMAGE=docker.io/perconalab/percona-server-mongodb-operator:PR-2058-78121c567 IMAGE_MONGOD=docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 IMAGE_MONGOD_CHAIN=docker.io/perconalab/percona-server-mongodb-operator:main-mongod6.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod7.0 docker.io/perconalab/percona-server-mongodb-operator:main-mongod8.0 IMAGE_SEARCH=perconalab/percona-server-mongodb-operator:main-mongot IMAGE_BACKUP=docker.io/perconalab/percona-server-mongodb-operator:main-backup IMAGE_PMM_CLIENT=docker.io/percona/pmm-client:2.44.1-1 IMAGE_PMM_SERVER=docker.io/perconalab/pmm-server:dev-latest IMAGE_PMM3_CLIENT=docker.io/perconalab/pmm-client:3-dev-latest IMAGE_PMM3_SERVER=docker.io/perconalab/pmm-server:3-dev-latest IMAGE_LOGCOLLECTOR=docker.io/perconalab/fluentbit:main-logcollector IMAGE_CLUSTERSYNC=docker.io/percona/percona-clustersync-mongodb:0.9.0 CERT_MANAGER_VER=1.19.3 CHAOS_MESH_VER=2.7.1 MINIO_VER=5.4.0 PMM_SERVER_VER=9.9.9 CLEAN_NAMESPACE=1 DELETE_CRD_ON_START=1 SKIP_DELETE=0 SKIP_BACKUPS_TO_AWS_GCP_AZURE= UPDATE_COMPARE_FILES=0 - generated xml file: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/reports/demand-backup-sharded.xml - - Generated html report: file:///mnt/jenkins/workspace/cloud-psmdb-operator_PR-2058/e2e-tests/reports/demand-backup-sharded.html - =========================== short test summary info ============================ FAILED e2e-tests/test_pytest_wrapper.py::test_bash_wrapper[demand-backup-sharded] - Failed: Test demand-backup-sharded failed with exit code 1 ======================== 1 failed in 1905.98s (0:31:45) ========================