Configure the cluster

Set up an HSTS HA cluster by configuring HSTS, the Redis database, and Redis Sentinel on each node. Configure one node as the Redis primary node, and the others as Redis replica nodes.

About this task

The following procedure describes the configuration steps for an HA cluster. For more information about Redis technology, see Redis documentation. (V.4.4.5).

These instructions use the default values for the Redis and Sentinel ports (31415 and 41415). If you use other ports, be sure that you adjust the commands and configuration files.

Important: The following documentation includes illustrative configuration examples; however, specifying passwords in the command line is not a secure practice.

Attention: To avoid issues during the configuration, always restart the involved services after every change you make. For more information, see Start the cluster.

Procedure

On each node, stop any HSTS transfers that are in process, or allow them to finish.

On each node, stop the HSTS processes that use Redis:

# systemctl stop asperanoded 
 # systemctl stop asperaredisd
 # systemctl stop asperarund

On each node, configure HSTS for Redis.
The following commands add the necessary configuration elements to the aspera.conf file.
1. Set the database type to Redis:
```
# asconfigurator -x "set_server_data;db_type,ha_redis"
```
2. Configure the private IP addresses and port numbers for each of the nodes in the clusters so Sentinel can communicate with them:
  - Define the IP addresses and ports in comma-separated lists.
  - Match the order of the IP addresses and ports for each node.
  - Use the same port numbers when you configure Redis Sentinel in a later step.
  Use the following syntax to set IP addresses:
```
# asconfigurator -x 'set_server_data;db_host,"node1-ip,node2-ip,node3-ip"'
 
```
  Use the following syntax to set the port numbers:
```
# asconfigurator -x 'set_server_data;db_port,"node1-port,node2-port,node3-port"'
```
  For example,
```
# asconfigurator -x 'set_server_data;db_host,"10.11.100.160,10.11.52.248,10.11.52.249"'
 # asconfigurator -x 'set_server_data;db_port,"41415,41415,41415"'
```
  Replace the values in the example with your own port numbers. If you want to use a port other than 41415, you must use the same port number for Sentinel, aspera.conf, and the firewall.
3. Provide your own name to the Redis myprimary_set instead of using the default name.
  A primary set defines the name of a cluster monitored by Redis Sentinel. By default, Redis Sentinel names the cluster myprimary. To set an alternative name, use the following syntax:
```
# asconfigurator -x "set_server_data;db_primaryset,myprimary_set" 
```
  The value assigned to db_primaryset can include A-z, 0-9, and ".", "-", or "_". The name cannot include special characters or spaces.
  Use the same name, the default or the one you set, when you configure Redis Sentinel itself.
4. Verify that /opt/aspera/etc/aspera.conf is properly configured. The <server> section looks like this example, but with the IP addresses for your nodes:
```
<server>
        <db_type>ha_redis</db_type>
        <db_host>10.11.100.160,10.11.52.248,10.11.52.249</db_host>
        <db_port>41415,41415,41415</db_port>
</server>
 
```

On the primary node, edit /opt/aspera/etc/redis/aspera_31415.conf.

Set bind to the private IP address of the primary node.

For example,


# Redis configuration for primary node

port 31415                 # Redis port
dbfilename redis.31415.rdb # Database file name
dir /opt/aspera/var        # Data directory
pidfile /opt/aspera/var/run/redis.31415.pid # PID file location

requirepass password       # Authentication password
masterauth password        # Password for replica authentication

save 60 10000              # Snapshot every 60 seconds if 10,000 keys change
daemonize no               # Managed by systemd
syslog-enabled yes         # Enable syslog logging
syslog-facility local2     # Syslog facility

bind 10.11.100.160         # Private IP address of this node
aof-use-rdb-preamble no    # Append-only file setting

Note the following directives in the configuration:

Keyword	Argument	Description
`port`	31415	Specifies the name of the database file.
`dbfilename`	redis.31415.rdb	Specifies the filename.
`dir`	/opt/aspera/var	Specifies its location.
`pidfile`	/opt/aspera/var/run/redis.31415.pid	Process Identifier. The operating system assigns a unique number to the Redis server process when it starts.
`requirepass`	`password`	The password of the Redis instance.
`masterauth`	`password`	The password of the Redis primary node.
`save`	`seconds changes`	Specifies the Redis persistence snapshotting for the database. In the previous example, the data set is saved every 60 seconds if at least 10000 keys changed.
`daemonize`	`yes` or `no`	Is set by the installer as required for the OS service manager (systemd it is set to no). Do not change it.
`syslog-enabled`	`yes` or `no`	Enable logging to the system logger.
`syslog-facility local2`	`local2`	Syslog facility. `local2` is one of the available syslog facilities. It is used to categorize and organize log messages.
`bind`	`ip_address_primary_node`	Specifies the network interfaces that the server listens on, allowing control over which IP address can connect.
`aof-use-rdb-preamble`	`yes` or `no`	Enables combining an RDB snapshot with AOF data during AOF rewrites.

For each of the Redis replica nodes, edit /opt/aspera/etc/redis/aspera_31415.conf.

Update the values for the following directives and leave the rest of the settings unchanged as the primary node:

Directive
bind	Set it to the private IP address of the local replica node.
replicaof	Set it the private IP address and Redis port of the primary node.
requirepass	Set it the password for the Redis replica instance.
masterauth	Set it the same password as the Redis replica.

For example,


# Redis configuration for replica node

port 31415                 # Redis port
dbfilename redis.31415.rdb # Database file name
dir /opt/aspera/var        # Data directory
pidfile /opt/aspera/var/run/redis.31415.pid # PID file location

requirepass password       # Authentication password for this replica
masterauth password        # Password to authenticate with primary

save 60 10000              # Snapshot every 60 seconds if 10,000 keys change
daemonize no               # Managed by systemd
syslog-enabled yes         # Enable syslog logging
syslog-facility local2     # Syslog facility

bind 10.11.52.348          # Private IP address of this replica node
replicaof 10.11.100.160 31415 # Primary node IP and Redis port

On every node, create the Sentinel configuration file and set the file permissions.
1. Create the file /opt/aspera/etc/redis/sentinel_41415.conf.
2. Make the file writable by asperadaemon (which runs asperaredissentinel).
```
# chown asperadaemon:aspadmins /opt/aspera/etc/redis/sentinel_41415.conf 
```

On the primary node, configure Redis Sentinel. The following instructions have the steps that you need to complete for the required configuration for your cluster.

Note: You can refer to /opt/aspera/etc/redis/sentinel.conf.template for a full set of directives and more information.

Add or update the following lines to the sentinel_41415.conf file:
- sentinel auth-pass myprimary password, where myprimary is the primary_name, and password is the Redis password. This allows Sentinel to authenticate to Redis.
- requirepass password, where requirepass sets a password for the Redis Sentinel, and the password must be the same as the one used for Redis authentication.
- Replace the IP address in the sentinel monitor line with the private IP of the primary node.

Use the following syntax to add your configuration information to /opt/aspera/etc/redis/sentinel_41415.conf:


# Redis Sentinel configuration

pidfile /opt/aspera/var/run/redis_sentinel_41415.pid  # PID file location
daemonize no                                          # Managed by systemd
dir /tmp                                              # Working directory
port 41415                                            # Sentinel port

bind private_ip_address                               # Private IP of this node

sentinel auth-pass primary_name password              # Password for Sentinel to authenticate with Redis
requirepass password                                  # Password for Sentinel clients

sentinel monitor primary_name primary_private_ip_address 31415 quorum  # Monitor primary node
sentinel down-after-milliseconds primary_name down_time                # Time before marking primary as down
sentinel failover-timeout primary_name timeout                         # Failover timeout

syslog-ident sentinel                                 # Syslog identity
syslog-enabled yes                                    # Enable syslog logging
syslog-facility local2                                # Syslog facility

The elements of the Redis configuration directives are as follows:

Keywords	Arguments	Description
`pidfile`	/opt/aspera/var/run/redis_sentinel.41415.pid	The Redis PID file.
`dir`	`/tmp`	Redis working directory.
`daemonize`	`Yes` or `No`	Must match the setting that you used in aspera_31415.conf.
`port`	`41415`	The port number used for Sentinel communication. The port number must match the one in aspera.conf and the firewall.
`bind`	`private_ip_address`	The private IP address of the local node.
`sentinel auth-pass`	`primary_name` and `password`	The name and the password of the primary node. It must match the name that is assigned to `db_primaryset`. Use `myprimary` if you are using the default name.
`requirepass`	`password`	Configuration directive used to enable password-based authentication for clients connecting to the Redis server.
`sentinel monitor`	`primary_name`	The name of the primary node. It must match the name that is assigned to `db_primaryset`. Use `myprimary` if you are using the default name. If you renamed it using asconfigurator (as described before), use that name. Use the private IP address of the primary node.
	`primary_ip_address`	The IP address of the primary node.
	31415	The port number that is used for Redis.
	`quorum`	The number of Redis Sentinel instances that must agree the primary is unreachable. This number must be less than the total number of Sentinel instances and at least 2.
`sentinel down-after-milliseconds`	`primary_name`	The name of the primary node. It must match the name that is assigned to `db_primaryset`. Use `myprimary` if you are using the default name. If you renamed it using asconfigurator (as described before), use that name.
	`down_time`	The number of milliseconds after which a node is reported as failing.
`sentinel failover-timeout`	`primary_name`	The name of the primary node. It must match the name that is assigned to `db_primaryset`. Use `myprimary` if you are using the default name. If you renamed it using asconfigurator (as described before), use that name.
	`timeout`	The number of milliseconds after which a failover process is started.
`syslog-ident`	`sentinel`	Syslog identity.
`syslog-enabled`	`yes`	Enable logging to the system logger.
`syslog-facility local2`	`local2`	Syslog facility.

The following example shows the configuration for a primary node:


# Redis Sentinel configuration for primary node

pidfile "/opt/aspera/var/run/redis_sentinel_41415.pid"  # PID file location
daemonize no                                            # Managed by systemd
dir "/tmp"                                              # Working directory
port 41415                                              # Sentinel port

bind 10.11.100.160                                      # Private IP address of this node

sentinel monitor myprimary 10.11.100.160 31415 2        # Monitor primary node (IP, Redis port, quorum)
sentinel down-after-milliseconds myprimary 300          # Time before marking primary as down
sentinel failover-timeout myprimary 1000                # Failover timeout in milliseconds

sentinel auth-pass myprimary password                   # Password for Sentinel to authenticate with Redis
requirepass password                                    # Password for Sentinel clients

syslog-ident sentinel                                   # Syslog identity
syslog-enabled yes                                      # Enable syslog logging
syslog-facility local2                                  # Syslog facility

On each of the replica nodes, configure Redis Sentinel:

Copy the contents of /opt/aspera/etc/redis/sentinel_41415.conf from the primary node to the replica node's sentinel_41415.conf file.

On each replica node, replace the bind directive with the private IP address of that replica node. The authentication lines stay the same (sentinel auth-pass myprimary password and requirepass password)

The following example shows a sentinel_41415.conf file for the replica nodes:


# Redis Sentinel configuration for replica node

pidfile "/opt/aspera/var/run/redis_sentinel_41415.pid"  # PID file location
daemonize no                                            # Managed by systemd
dir "/tmp"                                              # Working directory
port 41415                                              # Sentinel port

bind 10.11.52.248                                       # Private IP address of this replica node

sentinel monitor myprimary 10.11.100.160 31415 2        # Monitor primary node (IP, Redis port, quorum)
sentinel down-after-milliseconds myprimary 300          # Time before marking primary as down
sentinel failover-timeout myprimary 1000                # Failover timeout in milliseconds

sentinel auth-pass myprimary password                   # Password for Sentinel to authenticate with Redis
requirepass password                                    # Password for Sentinel clients

syslog-ident sentinel                                   # Syslog identity
syslog-enabled yes                                      # Enable syslog logging
syslog-facility local2                                  # Syslog facility

On all of the nodes (the primary and the replica nodes), enable the asperaredissentinel service.
Run the following commands:
```
# cp /opt/aspera/etc/redis/asperaredissentinel.service /lib/systemd/system/ 
 # systemctl enable asperaredissentinel
```
The enablement persists across restarts.
Set the password in redis.conf to allow clients to authenticate with the Redis database using askmscli. Do this step on all nodes (the primary and the replica nodes). (V4.4.5)

When the authorization layer is enabled, Redis blocks any requests from unauthenticated clients. A client can authenticate itself by sending the AUTH command followed by the password.
1. Export the Redis password to then add it to askmscli:
  
  The export requires the password used to access the Redis instance (the password is the same as the one set in redis.conf in step 4).
```
# export redis_pass=password
 # echo -n $redis_pass | sudo askmscli -s redis-password
```
2. For the user that runs asperanoded, set the keystore. The default user is asperadaemon. The username might change if you are not using the default user.
```
# sudo askmscli -i -u asperadaemon 
 # sudo askmscli -i -u xfer
```
3. For any transfer user, set up askmscli to make sure that they can use it with ascp and authenticate with Redis. Whether a transfer user is associated with an access key, initialize their keystore by running askmscli. The username is the transfer user's username.
  
  Repeat this step for every new transfer user added to the server.
```
 sudo askmscli -i -u username
```
Create firewall rules to allow only the required communication within the cluster between Redis replicas and the Redis primary, between Sentinel instances, and between asperanoded and Sentinel instances.
- In a nonclustered environment, the asperaredis process accepts connections only from the loopback address 127.0.0.1.
- In a clustered environment, Redis and Sentinel are configured to bind to private IP addresses. To improve security, after setting Redis to listen on private IP addresses, connections must only be allowed from other machines in the HSTS or Redis HA cluster. This prevents external access to Redis and Sentinel ports.
For example, assuming the cluster nodes use:
- Primary: 10.11.100.160
- Replica: 10.11.52.248
- Second replica: 10.11.52.249)
Run these commands on each node:
```
# Allow Sentinel traffic only from cluster IPs   
iptables -A INPUT -p tcp -s 10.11.100.160,10.11.52.248,10.11.52.249 --dport 41415 -j ACCEPT
 # Allow Redis traffic only from cluster IPs
iptables -A INPUT -p tcp -s 10.11.100.160,10.11.52.248,10.11.52.249 --dport 31415 -j ACCEPT
```
Then, check the port configuration on each node:
```
# iptables -L -n
```
Important: Test these rules in your environment to ensure they work as expected. Persist the rules across restarts by using your OS-specific firewall management tool (for example, iptables-save or firewalld).

Start the cluster.

On the primary node, start asperaredisd, asperaredissentinel, asperanoded, and asperarund:

# systemctl start asperaredisd
 # systemctl start asperaredissentinel
 # systemctl start asperanoded
 # systemctl start asperarund

On each of the replica nodes, use the same commands:

# systemctl start asperaredisd
 # systemctl start asperaredissentinel
 # systemctl start asperanoded
 # systemctl start asperarund

On each node, verify that the services are running:

# systemctl status asperaredisd
 # systemctl status asperaredissentinel
 # systemctl status asperanoded
 # systemctl status asperarund

Test the cluster for a successful failover.
1. Enable the DEBUG commands on Redis by adding enable-debug-command yes to the aspera_31415.conf file on the primary node.
2. Then, run:
```
# /opt/aspera/bin/asredis -p redis_port -h node_ip -a redis_password DEBUG sleep num_seconds
```
  Where redis_port is the port to connect to, -h node_ip is the IP address of the Redis node to connect to, -a redis_password is the Redis password, and DEBUG sleep num_seconds is the debugging command that pauses Redis for a specified number of seconds.
3. On any of the other nodes, use the following command to identify the current primary node:
```
# /opt/aspera/bin/asredis -p sentinel_port -h node_ip -a sentinel_password SENTINEL get-master-addr-by-name myprimary_name
```
Verify that the replication is working correctly across the clusters:
1. From one node, create a user:
```
# /opt/aspera/bin/asnodeadmin -au usr2 -p pass2 -x root
```
2. From a second node, check that the user was created. Run:
```
# /opt/aspera/bin/asnodeadmin -l
```
3. From a third node, remove the user:
```
# /opt/aspera/bin/asnodeadmin -du usr2
```
4. From the first node, check that the user is deleted:
```
# /opt/aspera/bin/asnodeadmin -l
```