Setting up disaster recovery for Metric Anomaly Detection
Learn how to set up disaster recovery for Metric Anomaly Detection.
About this task
This script must be run on a Linux® host with the following tools installed (the tools are preinstalled or available on RHEL 8 hosts):
- bash
- curl
- jq
Procedure
- Run the steps mentioned in Kubernetes secret synchronization.
- Run the steps mentioned in Exposing Apache CouchDB.
- Save the script as couchdbReplication.sh on the local disk and set the
executable part as chmod u+x couchdbReplication.sh.
#!/bin/bash #/************************************************* {COPYRIGHT-TOP} **** #* Licensed Materials - Property of IBM #* #* "Restricted Materials of IBM" #* #* © Copyright IBM Corp. 2022 All Rights Reserved. #* #* 5725-Q09 #* #* US Government Users Restricted Rights - Use, duplication, or #* disclosure restricted by GSA ADP Schedule Contract with IBM Corp. #************************************************** {COPYRIGHT-END} ****/ # This script can be used to configure a bidirectional replication between two sides of RBA. # If variables are exported and valid, the script will not prompt for values, otherwise an # interactive wizard will prompt for missing values. # Credentials can be defined individually per side, if they change, the setup needs to be repeated. # Headless mode: # export RBA_DB_URL_SIDE_A="https://rba-geo-1.company.com:6984" # export RBA_DB_USER_SIDE_A="xxx" # export RBA_DB_PW_SIDE_A="xxx" # export RBA_DB_URL_SIDE_B="https://rba-geo-2.company.com:6984" # export RBA_DB_USER_SIDE_B="xxx" # export RBA_DB_PW_SIDE_B="xxx" # ./couchdbReplication.sh setup|delete ACTION="${1}" # ensure only allowed actions are defined if [[ "${ACTION}" != "setup" ]] && [[ "${ACTION}" != "delete" ]]; then echo "Usage: ./couchdbReplication.sh setup|delete" exit 1 fi set -e echo "================ RBA Replication Setup ================" echo "Running in ${ACTION} mode" # validate that prerequisites are available if ! command -v jq &> /dev/null; then echo "jq could not be found" exit fi if ! command -v curl &> /dev/null; then echo "curl could not be found" exit fi ###################################################################### # side A config setup while true; do if [[ -z "${RBA_DB_URL_SIDE_A}" ]]; then echo -n "CouchDB URL side A: " read -r RBA_DB_URL_SIDE_A if [[ -z "${RBA_DB_URL_SIDE_A}" ]]; then echo "CouchDB URL cannot be empty" continue fi fi if [[ -z "${RBA_DB_USER_SIDE_A}" ]]; then echo -n "CouchDB username side A: " read -r RBA_DB_USER_SIDE_A if [[ -z "${RBA_DB_USER_SIDE_A}" ]]; then echo "CouchDB username cannot be empty" continue fi fi if [[ -z "${RBA_DB_PW_SIDE_A}" ]]; then echo -n "CouchDB password side A: " read -rs RBA_DB_PW_SIDE_A echo "" if [[ -z "${RBA_DB_PW_SIDE_A}" ]]; then echo "CouchDB password cannot be emtpy" continue fi fi # side A config validation set +e RBA_COUCHDB_STATUS=$(curl -k -sL -w "%{http_code}\\n" -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" "${RBA_DB_URL_SIDE_A}" -o /dev/null) set -e if [[ "200" == "${RBA_COUCHDB_STATUS}" ]]; then echo "CouchDB on side A is accessible" # the check if databases are initialized is only important to setup a replication, not to delete it if [[ "${ACTION}" == "setup" ]] ; then echo "Detecting runbook automation databases" RBA_COUCHDB_ALL_DBS=$(curl -k -s -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" "${RBA_DB_URL_SIDE_A}/_all_dbs" | jq -r '.[]') # only include dbs with the string rba (excludes _global_changes, _replicator and _users or other potential non-rba dbs) # RBA_DB_NAMES=$(echo "${RBA_COUCHDB_ALL_DBS}" | jq -r '.[] | select( . | test("(rba)"))') RBA_DB_NAMES="" for CURRENT_DB_NAME in ${RBA_COUCHDB_ALL_DBS}; do if [[ ("${CURRENT_DB_NAME}" == *"rba"* && "${CURRENT_DB_NAME}" != "rba-pdoc") || "${CURRENT_DB_NAME}" == "collabopsuser" ]]; then echo "replicate: ${CURRENT_DB_NAME}" RBA_DB_NAMES="${RBA_DB_NAMES} ${CURRENT_DB_NAME}" else echo "skip: ${CURRENT_DB_NAME}" fi done for CURRENT_DB_NAME in ${RBA_DB_NAMES}; do echo "RBA db detected: ${CURRENT_DB_NAME}" if [[ "${CURRENT_DB_NAME}" == "collabopsuser" ]]; then break fi set +e RBA_DESIGN_DOC_STATUS=$(curl -k -sL -w "%{http_code}\\n" -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" "${RBA_DB_URL_SIDE_A}/${CURRENT_DB_NAME}/_design/georedundancy" -o /dev/null) set -e if [[ "200" == "${RBA_DESIGN_DOC_STATUS}" ]]; then echo "CouchDB design doc ${RBA_DB_URL_SIDE_A}/${CURRENT_DB_NAME}/_design/georedundancy is accessible" else echo "CouchDB design doc ${RBA_DB_URL_SIDE_A}/${CURRENT_DB_NAME}/_design/georedundancy is not accessible" echo "Please ensure that the IBM Runbook Automation database is initialized on side A and rerun this script" exit 1 fi done fi break; else echo "CouchDB could not be accessed: ${RBA_COUCHDB_STATUS}" # at least one of the following variables are incorrect, # resetting them to trigger new prompt on next loop execution unset RBA_DB_URL_SIDE_A RBA_DB_USER_SIDE_A RBA_DB_PW_SIDE_A fi done ###################################################################### # side B config setup while true; do if [[ -z "${RBA_DB_URL_SIDE_B}" ]]; then echo -n "CouchDB URL side B: " read -r RBA_DB_URL_SIDE_B if [[ -z "${RBA_DB_URL_SIDE_B}" ]]; then echo "CouchDB URL cannot be emtpy" continue fi fi if [[ -z "${RBA_DB_USER_SIDE_B}" ]]; then echo -n "CouchDB username side B: " read -r RBA_DB_USER_SIDE_B if [[ -z "${RBA_DB_USER_SIDE_B}" ]]; then echo "CouchDB username cannot be emtpy" continue fi fi if [[ -z "${RBA_DB_PW_SIDE_B}" ]]; then echo -n "CouchDB password side B: " read -rs RBA_DB_PW_SIDE_B echo "" if [[ -z "${RBA_DB_PW_SIDE_B}" ]]; then echo "CouchDB password cannot be emtpy" continue fi fi # side B config validation set +e RBA_COUCHDB_STATUS=$(curl -k -sL -w "%{http_code}\\n" -u "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" "${RBA_DB_URL_SIDE_B}" -o /dev/null) set -e if [[ "200" == "${RBA_COUCHDB_STATUS}" ]]; then echo "CouchDB on side B is accessible" # Do not check if side B is initialized as the replication can create targets if necessary. # This will simplify the setup after a disaster. break; else echo "CouchDB could not be accessed: ${RBA_COUCHDB_STATUS}" # at least one of the following variables are incorrect, # resetting them to trigger new prompt on next loop execution unset RBA_DB_URL_SIDE_B RBA_DB_USER_SIDE_B RBA_DB_PW_SIDE_B fi done if [[ "${ACTION}" == "setup" ]] ; then ###################################################################### # computing basic auth headers required for replication documents AUTH_HEADER_SIDE_A="Basic $(echo -ne "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" | base64)" AUTH_HEADER_SIDE_B="Basic $(echo -ne "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" | base64)" ###################################################################### echo "Creating replication for Runbook Service Database" for CURRENT_DB_NAME in ${RBA_DB_NAMES}; do echo "Setting up replication: ${RBA_DB_URL_SIDE_A}/${CURRENT_DB_NAME} -> ${RBA_DB_URL_SIDE_B}/${CURRENT_DB_NAME}" echo "Deleting old replication if present, continuing if missing" # get old rev to delete document OLD_REV=$(curl \ -k \ -s \ -X GET \ -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" \ "${RBA_DB_URL_SIDE_A}/_replicator/${CURRENT_DB_NAME}-georedundancy-replication" | jq -r "._rev") # delete document curl \ -k \ -s \ -X DELETE \ -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" \ "${RBA_DB_URL_SIDE_A}/_replicator/${CURRENT_DB_NAME}-georedundancy-replication?rev=${OLD_REV}" # create document on side A pushing docs to side B if [[ "${CURRENT_DB_NAME}" == "collabopsuser" ]]; then echo "Creating replication" CREATE_REPLICATION_RBA_A_TO_B=$(curl \ -k \ -s \ -X POST \ -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" \ -H "Content-Type: application/json" \ "${RBA_DB_URL_SIDE_A}/_replicator" -d ' { "_id": "'"${CURRENT_DB_NAME}"'-georedundancy-replication", "source": { "url": "'"${RBA_DB_URL_SIDE_A}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_A}"'" } }, "target": { "url": "'"${RBA_DB_URL_SIDE_B}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_B}"'" } }, "create_target": true, "continuous": true }') else echo "Creating replication" CREATE_REPLICATION_RBA_A_TO_B=$(curl \ -k \ -s \ -X POST \ -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" \ -H "Content-Type: application/json" \ "${RBA_DB_URL_SIDE_A}/_replicator" -d ' { "_id": "'"${CURRENT_DB_NAME}"'-georedundancy-replication", "source": { "url": "'"${RBA_DB_URL_SIDE_A}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_A}"'" } }, "target": { "url": "'"${RBA_DB_URL_SIDE_B}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_B}"'" } }, "filter": "georedundancy/standard", "create_target": true, "continuous": true }') fi # expected result: {"ok":true,"id":"rba-georedundancy-replication","rev":"1-9c73161d3485230f84ec12f0c922f5a3"} echo "${CREATE_REPLICATION_RBA_A_TO_B}" if [[ "$(echo "${CREATE_REPLICATION_RBA_A_TO_B}" | jq ".ok")" ]] ; then echo "Replication for ${CURRENT_DB_NAME} from side A to B was configured successfully" else echo "Failed to setup replication for ${CURRENT_DB_NAME} from side A to B" exit 1 fi ###################################################################### echo "${RBA_DB_URL_SIDE_B}/${CURRENT_DB_NAME} -> ${RBA_DB_URL_SIDE_A}/${CURRENT_DB_NAME}" echo "Deleting old replication if present, continuing if missing" # get old rev to delete document OLD_REV=$(curl \ -k \ -s \ -X GET \ -u "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" \ "${RBA_DB_URL_SIDE_B}/_replicator/${CURRENT_DB_NAME}-georedundancy-replication" | jq -r "._rev") # delete document curl \ -k \ -s \ -X DELETE \ -u "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" \ "${RBA_DB_URL_SIDE_B}/_replicator/${CURRENT_DB_NAME}-georedundancy-replication?rev=${OLD_REV}" # create document on side B pushing docs to side A echo "Creating replication document" CREATE_REPLICATION_RBA_B_TO_A=$(curl \ -k \ -s \ -X POST \ -u "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" \ -H "Content-Type: application/json" \ "${RBA_DB_URL_SIDE_B}/_replicator" -d ' { "_id": "'"${CURRENT_DB_NAME}"'-georedundancy-replication", "source": { "url": "'"${RBA_DB_URL_SIDE_B}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_B}"'" } }, "target": { "url": "'"${RBA_DB_URL_SIDE_A}"'/'"${CURRENT_DB_NAME}"'", "headers": { "Authorization": "'"${AUTH_HEADER_SIDE_A}"'" } }, "filter": "georedundancy/standard", "create_target": true, "continuous": true }') echo "${CREATE_REPLICATION_RBA_B_TO_A}" # expected result: {"ok":true,"id":"rba-georedundancy-replication","rev":"1-9c73161d3485230f84ec12f0c922f5a3"} echo "${CREATE_REPLICATION_RBA_B_TO_A}" if [[ "$(echo "${CREATE_REPLICATION_RBA_B_TO_A}" | jq ".ok")" ]] ; then echo "Replication for ${CURRENT_DB_NAME} from side B to A was configured successfully" else echo "Failed to setup replication for ${CURRENT_DB_NAME} from side B to A" exit 1 fi done echo "Replication setup completed, displaying replication status" else function deleteRbaReplications { CURRENT_DB_SERVER=${1} CURRENT_DB_USER=${2} CURRENT_DB_PW=${3} echo "Detecting runbook automation replications on ${CURRENT_DB_SERVER}" COUCHDB_REPLICATIONS=$(curl -k -s -u "${CURRENT_DB_USER}:${CURRENT_DB_PW}" "${CURRENT_DB_SERVER}/_replicator/_all_docs") # check for rba and georedundancy-replication in the doc id to delete # potential doc names would be # bmstage-30363937666435352d393238372d343836382d383365362d353832393337313031313137-rba-as-georedundancy-replication # bmstage-30363937666435352d393238372d343836382d383365362d353832393337313031313137-rba-rbs-georedundancy-replication # rba-as-georedundancy-replication # rba-georedundancy-replication RBA_COUCHDB_REPLICATION_DOCS=$(echo "${COUCHDB_REPLICATIONS}" | jq -r '.rows[].id | select( . | test("(.*georedundancy-replication)"))') if [[ "${RBA_COUCHDB_REPLICATION_DOCS}" == "" ]]; then echo "No RBA replications were detected on ${CURRENT_DB_SERVER}, skipping deletion" return fi for CURRENT_DOC_TO_DELETE in ${RBA_COUCHDB_REPLICATION_DOCS}; do echo "Deleting replication doc ${CURRENT_DOC_TO_DELETE} on ${CURRENT_DB_SERVER}" # fetch rev required to delete replication doc CURRENT_DOC_REV=$(curl \ -k \ -s \ -X GET \ -u "${CURRENT_DB_USER}:${CURRENT_DB_PW}" \ "${CURRENT_DB_SERVER}/_replicator/${CURRENT_DOC_TO_DELETE}" | jq -r "._rev") # delete replication document DELETE_REPLICATION_DOC=$(curl \ -k \ -s \ -X DELETE \ -u "${CURRENT_DB_USER}:${CURRENT_DB_PW}" \ "${CURRENT_DB_SERVER}/_replicator/${CURRENT_DOC_TO_DELETE}?rev=${CURRENT_DOC_REV}") echo "${DELETE_REPLICATION_DOC}" if [[ "$(echo "${DELETE_REPLICATION_DOC}" | jq ".ok")" ]] ; then echo "Replication ${CURRENT_DOC_TO_DELETE} on ${CURRENT_DB_SERVER} was delete successfully" else echo "Failed to delete replication doc ${CURRENT_DOC_TO_DELETE} on ${CURRENT_DB_SERVER}" exit 1 fi done } deleteRbaReplications "${RBA_DB_URL_SIDE_A}" "${RBA_DB_USER_SIDE_A}" "${RBA_DB_PW_SIDE_A}" deleteRbaReplications "${RBA_DB_URL_SIDE_B}" "${RBA_DB_USER_SIDE_B}" "${RBA_DB_PW_SIDE_B}" fi echo "Showing current CouchDB scheduler jobs" echo "Side A:" curl \ -k \ -s \ -X GET \ -u "${RBA_DB_USER_SIDE_A}:${RBA_DB_PW_SIDE_A}" \ "${RBA_DB_URL_SIDE_A}/_scheduler/jobs" | jq "." echo "Side B:" curl \ -k \ -s \ -X GET \ -u "${RBA_DB_USER_SIDE_B}:${RBA_DB_PW_SIDE_B}" \ "${RBA_DB_URL_SIDE_B}/_scheduler/jobs" | jq "." - To set up the replication, run the following command and enter the prompted values:
$ ./couchdbReplication.sh Usage: ./couchdbReplication.sh setup|deleteExample:./couchdbReplication.sh setup ================ RBA Replication Setup ================ Running in setup mode CouchDB on side A is accessible Detecting runbook automation databases skip: _global_changes skip: _replicator skip: _users replicate: collabopsuser skip: emailrecipients skip: genericproperties replicate: icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as replicate: icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs skip: integration skip: noi-aggconfigdb skip: noi-osregdb skip: osb-map skip: otc_omaas_broker skip: rba-pdoc skip: schedule skip: tenant skip: trainingjob RBA db detected: collabopsuser CouchDB on side B is accessible Creating replication for Runbook Service Database Setting up replication: https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/collabopsuser -> https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/collabopsuser Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication {"ok":true,"id":"collabopsuser-georedundancy-replication","rev":"1-684fa9c5ca9b0b04b9df0fecb8988871"} Replication for collabopsuser from side A to B was configured successfully https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/collabopsuser -> https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/collabopsuser Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication document {"ok":true,"id":"collabopsuser-georedundancy-replication","rev":"1-e42ef5e23a59391eeb3763aeedb39976"} {"ok":true,"id":"collabopsuser-georedundancy-replication","rev":"1-e42ef5e23a59391eeb3763aeedb39976"} Replication for collabopsuser from side B to A was configured successfully Setting up replication: https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as -> https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication","rev":"1-db0a27c8a7d3b7332b3ebb3f3e046fc0"} Replication for icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as from side A to B was configured successfully https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as -> https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication document {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication","rev":"1-ee9081a2c823e903a82fac497e78a493"} {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication","rev":"1-ee9081a2c823e903a82fac497e78a493"} Replication for icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as from side B to A was configured successfully Setting up replication: https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs -> https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication","rev":"1-f5f63f21fa090323d03eaafa694c03a9"} Replication for icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs from side A to B was configured successfully https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs -> https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs Deleting old replication if present, continuing if missing {"error":"not_found","reason":"missing"} Creating replication document {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication","rev":"1-54a4fcf9dc1df30faa0703f4792ed8d4"} {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication","rev":"1-54a4fcf9dc1df30faa0703f4792ed8d4"} Replication for icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs from side B to A was configured successfully Replication setup completed, displaying replication status Showing current CouchDB scheduler jobs Side A: { "total_rows": 3, "offset": 0, "jobs": [ { "database": "_replicator", "id": "25b7ea12b4314ef19453ca55780a3f58+continuous+create_target", "pid": "<0.10373.3>", "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "user": null, "doc_id": "icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication", "history": [ { "timestamp": "2022-09-22T08:21:28Z", "type": "started" }, { "timestamp": "2022-09-22T08:21:28Z", "type": "added" } ], "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "start_time": "2022-09-22T08:21:28Z" }, { "database": "_replicator", "id": "30e190747c0f1b34d64235fb8b28f74c+continuous+create_target", "pid": "<0.9905.3>", "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as/", "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as/", "user": null, "doc_id": "icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication", "history": [ { "timestamp": "2022-09-22T08:21:20Z", "type": "started" }, { "timestamp": "2022-09-22T08:21:20Z", "type": "added" } ], "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "start_time": "2022-09-22T08:21:20Z" }, { "database": "_replicator", "id": "bfc5226c667b0727ab07323b99184e00+continuous+create_target", "pid": "<0.9526.3>", "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/collabopsuser/", "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/collabopsuser/", "user": null, "doc_id": "collabopsuser-georedundancy-replication", "history": [ { "timestamp": "2022-09-22T08:21:13Z", "type": "started" }, { "timestamp": "2022-09-22T08:21:13Z", "type": "added" } ], "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "start_time": "2022-09-22T08:21:13Z" } ] } Side B: { "total_rows": 1, "offset": 0, "jobs": [ { "database": "_replicator", "id": "afcde8d15fa0f9ee1420e7db24bdcf03+continuous+create_target", "pid": "<0.20967.2>", "source": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "target": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "user": null, "doc_id": "icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication", "history": [ { "timestamp": "2022-09-22T08:21:32Z", "type": "started" }, { "timestamp": "2022-09-22T08:21:32Z", "type": "added" } ], "node": "couchdb@backup-couchdb-0.backup-couchdb.backup.svc.cluster.local", "start_time": "2022-09-22T08:21:32Z" } ] }Alternatively, you can export variables to start the script in noninteractive mode. This step is useful when you start the script by automation:
$ export RBA_DB_URL_SIDE_A="https://couchdb.apps.noi-side-a.company.com" $ export RBA_DB_USER_SIDE_A="usera" $ export RBA_DB_PW_SIDE_A="xxx" $ export RBA_DB_URL_SIDE_B="https://couchdb.apps.noi-side-b.company.com" $ export RBA_DB_USER_SIDE_B="userb" $ export RBA_DB_PW_SIDE_B="xxx" $ ./couchdbReplication.sh setup - To stop the replication between sides, use the same script with the delete command (the
example uses the noninteractive
mode): Example:
./couchdbReplication.sh delete ================ RBA Replication Setup ================ Running in delete mode CouchDB on side A is accessible CouchDB on side B is accessible Detecting runbook automation replications on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com Deleting replication doc collabopsuser-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com {"ok":true,"id":"collabopsuser-georedundancy-replication","rev":"2-6fcb9facc438ce8e968a869df56420cc"} Replication collabopsuser-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com was delete successfully Deleting replication doc icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication","rev":"2-54a248efe3564d62d683ef812d700d1b"} Replication icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com was delete successfully Deleting replication doc icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication","rev":"2-966b8d8b123e67b3cd5baaa579de9716"} Replication icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication on https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com was delete successfully Detecting runbook automation replications on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com Deleting replication doc collabopsuser-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com {"ok":true,"id":"collabopsuser-georedundancy-replication","rev":"2-9aa83ba8ba017f635c52729bb21f8038"} Replication collabopsuser-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com was delete successfully Deleting replication doc icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication","rev":"2-cea95dfea595e4c738ead2a62ee53eb3"} Replication icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com was delete successfully Deleting replication doc icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com {"ok":true,"id":"icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication","rev":"2-60828d548a5a3df447d107767fc01d5f"} Replication icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication on https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com was delete successfully Showing current CouchDB scheduler jobs Side A: { "total_rows": 0, "offset": 0, "jobs": [] } Side B: { "total_rows": 0, "offset": 0, "jobs": [] }Note: If one side becomes unavailable for a short time, for example due to Kubernetes rescheduling actions or a router restart, the replication resumes automatically. If there is a longer outage, resume replication manually by running the replication setup again. The step deletes existing replication records and creates them again, which triggers a new replication on CouchDB instantaneously. - To verify whether the replication completed, run the following command against both
sides:
$ curl -sku xxx:xxx https://couchdb.apps.noi-side-a.company.com/_active_tasks | jq "."Example output from Curl:[ { "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "pid": "<0.10373.3>", "changes_pending": 0, "checkpoint_interval": 30000, "checkpointed_source_seq": "25-g1AAAAVJeJzN00sKwjAQBuCgBRUPoRewpNWmuFLwIDrJJFSJVVoruNKb6E30JnqTmtqKj6VU6GbCzCL_x5BoQkgzqCOZiFUiAuTjdTRfQrTrFX3Ptb8mz96Ot8IWOok3MrL1SoDW5rIaEN5J03QR1KG9NIOGdAcwdNxyIkql8q6pfFRoyUOrkDNF-0haSYhSzUOJ5WaOs8xpkVl7ZKKUEvmwihuaZdr9h9YZAKV9VUFtaJlKDuYw4ONL7HngIYXKik-5-JyJrfzP-B5FVt0dX3Lx9bVjxqWvzDP-v9j5SXzLxW_vmIIvFGPlJC3uoCjdEw", "continuous": true, "database": "shards/e0000000-ffffffff/_replicator.1663550895", "doc_id": "icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs-georedundancy-replication", "doc_write_failures": 0, "docs_read": 0, "docs_written": 0, "missing_revisions_found": 0, "replication_id": "25b7ea12b4314ef19453ca55780a3f58+continuous+create_target", "revisions_checked": 1, "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "source_seq": "25-g1AAAAVJeJzN00sKwjAQBuCgBRUPoRewpNWmuFLwIDrJJFSJVVoruNKb6E30JnqTmtqKj6VU6GbCzCL_x5BoQkgzqCOZiFUiAuTjdTRfQrTrFX3Ptb8mz96Ot8IWOok3MrL1SoDW5rIaEN5J03QR1KG9NIOGdAcwdNxyIkql8q6pfFRoyUOrkDNF-0haSYhSzUOJ5WaOs8xpkVl7ZKKUEvmwihuaZdr9h9YZAKV9VUFtaJlKDuYw4ONL7HngIYXKik-5-JyJrfzP-B5FVt0dX3Lx9bVjxqWvzDP-v9j5SXzLxW_vmIIvFGPlJC3uoCjdEw", "started_on": 1663834891, "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/", "through_seq": "25-g1AAAAVJeJzN00sKwjAQBuCgBRUPoRewpNWmuFLwIDrJJFSJVVoruNKb6E30JnqTmtqKj6VU6GbCzCL_x5BoQkgzqCOZiFUiAuTjdTRfQrTrFX3Ptb8mz96Ot8IWOok3MrL1SoDW5rIaEN5J03QR1KG9NIOGdAcwdNxyIkql8q6pfFRoyUOrkDNF-0haSYhSzUOJ5WaOs8xpkVl7ZKKUEvmwihuaZdr9h9YZAKV9VUFtaJlKDuYw4ONL7HngIYXKik-5-JyJrfzP-B5FVt0dX3Lx9bVjxqWvzDP-v9j5SXzLxW_vmIIvFGPlJC3uoCjdEw", "type": "replication", "updated_on": 1663834981, "user": null }, { "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "pid": "<0.9526.3>", "changes_pending": 0, "checkpoint_interval": 30000, "checkpointed_source_seq": "30-g1AAAAWDeJzN0ksKwjAQBuBgBV14B72AJU2jNisLHkQzaUqV-KBawZXeRG-iN9Gb1PQhPhaFQpFuJswsMh8_oxBC7cDw0ESsIxF44G7C-ZKHh37e94n5M3n15nYvTKGi7U6GploLrpT-rMERdOM4XgQGbyz1oIUdnwIl1awopOKSVOjpCuNca6RaJhwLpKxmS7VaN9FOv7IlYGGG2R-0Zc8AZon2-JWtBURK4tTwElZNXdFJPxp8fufrS0YHAmqYbya-ZOLrO2PsSJsyr7YZ3zLxPRF3UvHIt5k1pLXN-JGJP-7Y5twGUnQVJTYtnhJH8hM", "continuous": true, "database": "shards/e0000000-ffffffff/_replicator.1663550895", "doc_id": "collabopsuser-georedundancy-replication", "doc_write_failures": 0, "docs_read": 1, "docs_written": 1, "missing_revisions_found": 1, "replication_id": "bfc5226c667b0727ab07323b99184e00+continuous+create_target", "revisions_checked": 23, "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/collabopsuser/", "source_seq": "30-g1AAAAWDeJzN0ksKwjAQBuBgBV14B72AJU2jNisLHkQzaUqV-KBawZXeRG-iN9Gb1PQhPhaFQpFuJswsMh8_oxBC7cDw0ESsIxF44G7C-ZKHh37e94n5M3n15nYvTKGi7U6GploLrpT-rMERdOM4XgQGbyz1oIUdnwIl1awopOKSVOjpCuNca6RaJhwLpKxmS7VaN9FOv7IlYGGG2R-0Zc8AZon2-JWtBURK4tTwElZNXdFJPxp8fufrS0YHAmqYbya-ZOLrO2PsSJsyr7YZ3zLxPRF3UvHIt5k1pLXN-JGJP-7Y5twGUnQVJTYtnhJH8hM", "started_on": 1663834877, "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/collabopsuser/", "through_seq": "30-g1AAAAWDeJzN0ksKwjAQBuBgBV14B72AJU2jNisLHkQzaUqV-KBawZXeRG-iN9Gb1PQhPhaFQpFuJswsMh8_oxBC7cDw0ESsIxF44G7C-ZKHh37e94n5M3n15nYvTKGi7U6GploLrpT-rMERdOM4XgQGbyz1oIUdnwIl1awopOKSVOjpCuNca6RaJhwLpKxmS7VaN9FOv7IlYGGG2R-0Zc8AZon2-JWtBURK4tTwElZNXdFJPxp8fufrS0YHAmqYbya-ZOLrO2PsSJsyr7YZ3zLxPRF3UvHIt5k1pLXN-JGJP-7Y5twGUnQVJTYtnhJH8hM", "type": "replication", "updated_on": 1663834967, "user": null }, { "node": "couchdb@primary-couchdb-0.primary-couchdb.primary.svc.cluster.local", "pid": "<0.9905.3>", "changes_pending": 0, "checkpoint_interval": 30000, "checkpointed_source_seq": "13-g1AAAAUPeJzN01FqwjAYB_BghW2n0AtYkqw29mmCB5nJl4QqaSvVDva03URv4m7ibtKli2PVByFQii8J-R7y_5F8n0EIPaaBRAsoKkilmG_KVcbL98n5PKHhVeXvHG7fIARTbXeqDE0B3Bh72YAjMarrep0GfJDZwgORCWMEdxNxk4o9qWJsV_Fy1qJfbYIhilgs0VOVS6VXuZI3M4lv5rzJfL14oQhEokncTUqnnymWjfbjQou1ZAnVPfynrzYf2hV92s2C9_9iNWUg2PPdig9OfGzEgZuZWTylmvQg9u1fJ_5y4lOrh2NCqOA99LDvlDvxtxO3Jp1rhWkE7Ulf_wCeasTU", "continuous": true, "database": "shards/00000000-1fffffff/_replicator.1663550895", "doc_id": "icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as-georedundancy-replication", "doc_write_failures": 0, "docs_read": 1, "docs_written": 1, "missing_revisions_found": 1, "replication_id": "30e190747c0f1b34d64235fb8b28f74c+continuous+create_target", "revisions_checked": 1, "source": "https://couchdb-georedundancy-primary.apps.geor-1-166-180911.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as/", "source_seq": "13-g1AAAAUPeJzN01FqwjAYB_BghW2n0AtYkqw29mmCB5nJl4QqaSvVDva03URv4m7ibtKli2PVByFQii8J-R7y_5F8n0EIPaaBRAsoKkilmG_KVcbL98n5PKHhVeXvHG7fIARTbXeqDE0B3Bh72YAjMarrep0GfJDZwgORCWMEdxNxk4o9qWJsV_Fy1qJfbYIhilgs0VOVS6VXuZI3M4lv5rzJfL14oQhEokncTUqnnymWjfbjQou1ZAnVPfynrzYf2hV92s2C9_9iNWUg2PPdig9OfGzEgZuZWTylmvQg9u1fJ_5y4lOrh2NCqOA99LDvlDvxtxO3Jp1rhWkE7Ulf_wCeasTU", "started_on": 1663834883, "target": "https://couchdb-georedundancy-backup.apps.geor-2-166-180914.cp.xyz.com/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-as/", "through_seq": "13-g1AAAAUPeJzN01FqwjAYB_BghW2n0AtYkqw29mmCB5nJl4QqaSvVDva03URv4m7ibtKli2PVByFQii8J-R7y_5F8n0EIPaaBRAsoKkilmG_KVcbL98n5PKHhVeXvHG7fIARTbXeqDE0B3Bh72YAjMarrep0GfJDZwgORCWMEdxNxk4o9qWJsV_Fy1qJfbYIhilgs0VOVS6VXuZI3M4lv5rzJfL14oQhEokncTUqnnymWjfbjQou1ZAnVPfynrzYf2hV92s2C9_9iNWUg2PPdig9OfGzEgZuZWTylmvQg9u1fJ_5y4lOrh2NCqOA99LDvlDvxtxO3Jp1rhWkE7Ulf_wCeasTU", "type": "replication", "updated_on": 1663834973, "user": null } ] - For database migrations, disable replication, upgrade both sides, and then reactivate replication again. For more information, see Upgrading Runbook Automation in replicated environments.