Linux-UNIX: Couchbase auditing configuration

Use a double proxy to capture couchbase traffic with real client IPs.

About this task

This procedure requires two proxies of either Nginx or HAProxy type. Both must be located on the database.

Guidelines:

Incoming encrypted traffic from the Client is sent to the first proxy instance, Proxy1.
Proxy1 terminates the SSL and adds the proxy protocol to ports.
The second proxy instance, Proxy2, listens to intermediate ports and removes the proxy protocol.
K-TAP intercepts traffic in between Proxy1 and Proxy2 to get decrypted traffic with real client IPs.
Outgoing decrypted traffic from Proxy2 is sent to unecrypted Couchbase database ports.

Procedure

Configure the database.

These ports must be open on each host for Couchbase Server to operate correctly. In addition, these ports must be available (not blocked by a firewall or other such mechanism):

Between each node of a cluster
Between nodes of multiple clusters connected by using XDCR
Between application servers and nodes, and for administrative access

Port Name	Default Port Number Unencrypted / Encrypted	Description	Node-to-node	Client-to-Node	Cluster admin	XDCR v1 (CAPI)	XDCR v2 (XMEM)
rest_port / ssl_rest_port	8091 / 18091	REST/HTTP including web UI	Yes	Yes	Yes	Yes	Yes
capi_port / ssl_capi_port	8092 / 18092	Views and XDCR access	Yes	Yes	No	Yes	Yes
query_port / ssl_query_port	8093 / 18093	Query service REST/HTTP traffic	Yes	Yes	No	No	No
fts_http_port / fts_ssl_port	8094 / 18094	Search service REST/HTTP traffic	Yes	Yes	No	No	No
memcached_port / memcached_ssl_port	11210 / 11207	Data Service	Yes	Yes	No	No	Yes

Proxy Configuration: There are two ways to setup Proxy configurations along with IPTABLE configurations.
- Proxy1 listens on port 1xxxx, so external ports are still the same to the user. For example, an encrypted connection still connects via the GUI as http://COUCHBASE:1xxxx. In this case the couchbase configuration needs to make sure it is not listening on 1xxxx. Couchbase configuration is in the static_config file. To avoid reconfiguring clients, the default ports must be modified so that the clients connect to the first proxy instance instead. For example:
```
{memcached_ssl_port, 11207}
{ssl_rest_port, 18091}
{ssl_capi_port, 18092}
{ssl_query_port,18093}
{fts_ssl_port, 18094}
```
  These ports need to be used in the first proxy instance and Couchbase would need to be configured to expect unencrypted connections and to move these ports to a different number so that they do not duplicate ports being listened on by the proxy instances.
- Proxy1 listens on port yxxxx where y is not 1, so the couchbase configuration can stay as the default, but external ports need to be changed to yxxxx.
Configure HAProxy. (Not required if you are using NGINX.)
To enable Proxy Protocol in HAProxy1, add the send-proxy keyword to the /etc/haproxy/haproxy.cfg file. To get the Real Client IP, strip the Proxy protocol in HAProxy2, and add accept-proxy to /etc/haproxy/haproxy.cfg file.
Added/Modify haproxy.cfg for all couchbase ssl ports, giving an example of ssl port(1xxxx), where xxxx is SSL port, such as 8091/8092, etc; yyy is service name such as rest/capi/query, etc.
Method1: Proxy1 listens on port 1xxxx and pass traffic to Proxy2, Proxy2 listens on port 2xxxx and send decrypted traffic to couchbase no ssl port (xxxx). By using this method, couchbase should not listen on port 1xxxx.
- frontend yyy_in bind *:
```
bind *:1xxxx ssl crt /etc/haproxy/ha_couchbase_cert_key.pem
default_backend send_pp_yyy
```
- backend send_pp_yyy
```
balance roundrobin
server intermediate_yyy 127.0.0.1:2xxxx send-proxy
```
- frontend strip_pp_yyy
```
bind 127.0.0.1:2xxxx accept-proxy
default_backend yyy_out
```
- backend yyy_out
```
balance roundrobin
server yyyy 127.0.0.1:xxxx
```
Method2: Proxy1 listens on port 3xxxx and passes traffic to Proxy2. Proxy2 listens on port 2xxxx and sends decrypted traffic to couchbase no ssl port (xxxx). By using this method, the external port for user should be changed to 3xxxx.
- frontend yyy_in bind *:
```
bind *:3xxxx ssl crt /etc/haproxy/ha_couchbase_cert_key.pem
default_backend send_pp_yyy
```
- backend send_pp_yyy
```
balance roundrobin
server intermediate_yyy 127.0.0.1:2xxxx send-proxy
```
- frontend strip_pp_yyy
```
bind 127.0.0.1:2xxxx accept-proxy
default_backend yyy_out
```
- backend yyy_out
```
balance roundrobin
server yyyy 127.0.0.1:xxxx
```
Configure NGINX. (Not required if you are using HAProxy.)
For a TCP stream, the PROXY protocol can be enabled for connections between NGINX and an upstream server. To enable the PROXY protocol in Nginx1, include the proxy_protocol directive in a server block at the stream {} level in /etc/nginx/nginx.conf. NGINX terminates HTTPS traffic (the ssl_certificate and ssl_certificate_key directives) and proxies the decrypted data to a backend server. In Nginx2, add listen with proxy_protocol to receive the client’s real IP forwarded with Proxy Protocol.
1. Make sure that your NGINX installation includes the HTTP and Stream Real IP modules:
```
nginx -V 2>&1 | grep -- 'http_realip_module' 
nginx -V 2>&1 | grep -- 'stream_realip_module'
```
2. Add/modify nginx.cfg for all couchbase ssl ports, giving an example of ssl port(1xxxx), where xxxx is SSL port, such as 8091/8092, etc; yyy is service name such as rest/capi/query, etc.
  - Method1: Proxy1 listens on port 1xxxx and pass traffic to Proxy2, Proxy2 listens on port 2xxxx and sends decrypted traffic to couchbase no ssl port (xxxx). When using this method, couchbase should not listen on port 1xxxx.
```
stream {
    
   server {
       listen                  1xxxx ssl;
       proxy_pass              127.0.0.1:2xxxx;
       ssl_certificate         /opt/couchbase/SSLCA/chain.pem;
       ssl_certificate_key     /opt/couchbase/SSLCA/nodedir/pkey.key;
       ssl_session_cache       shared:SSL:8m;
       proxy_protocol          on;
   }
  
   server {
       listen                  2xxxx proxy_protocol;
       proxy_pass              127.0.0.1:xxxx;
   }
    
}
```
  - Method2: Proxy1 listens on port 3xxxx and pass traffic to Proxy2, Proxy2 listens on port 2xxxx and send decrypted traffic to couchbase no ssl port (xxxx). When using this method, the external port for user should be changed to 3xxxx.
```
stream {
    
   server {
       listen                  2xxxx ssl;
       proxy_pass              127.0.0.1:3xxxx;
       ssl_certificate         /opt/couchbase/SSLCA/chain.pem;
       ssl_certificate_key     /opt/couchbase/SSLCA/nodedir/pkey.key;
       ssl_session_cache       shared:SSL:8m;
       proxy_protocol          on;
   }
  
   server {
       listen                  3xxxx proxy_protocol;
       proxy_pass              127.0.0.1:xxxx;
   }
    
}
```

Configure the network.

If the client is on the same server with the DB, verify that all traffic from the client is sent to Proxy1. Otherwise, all traffic that is not from Proxy2 to DB is dropped. This requires IPTABLE setup.

In order to support encrypted traffic for couchbase, it is necessary to configure a proxy service to terminate the encryption. The proxy solution that is used depends on your requirements, though NGINX and HAProxy both work well for this. [Example configurations are attached above]

Using this script to setup IPTABLES. The UID can vary depending on your system configuration. As with all firewall rules, order is important.

#!/bin/sh

# Remove any existing jumps to our custom chains
iptables -D INPUT  -j chain-couchbase-incoming
iptables -D OUTPUT -j chain-couchbase-outgoing

# Clean any existing custom chains
iptables -F chain-couchbase-incoming
iptables -F chain-couchbase-outgoing
iptables -X chain-couchbase-incoming
iptables -X chain-couchbase-outgoing
iptables -N chain-couchbase-incoming
iptables -N chain-couchbase-outgoing

# Define external(proxy1), proxy2, ssl and no ssl ports
external_port_prefix=0
ssl_port_prefix=0
proxy2_port_prefix=0
port_list=0

# Pass in all arguments, please see bottom usage.
for i in "$@" ; do
  # set the port prefix
  if echo $i | grep '^-' > /dev/null; then
    if echo $i | grep '^-external_port_prefix$' > /dev/null; then
      external_port_prefix=999
    elif echo $i | grep '^-ssl_port_prefix$' > /dev/null; then
      ssl_port_prefix=999
    elif echo $i | grep '^-port_list$' > /dev/null; then
      port_list=999
    fi
  elif [ "X$external_port_prefix" = "X999" ]; then
    external_port_prefix=$i
  elif [ "X$proxy2_port_prefix" = "X999" ]; then
    proxy2_port_prefix=$i
  elif [ "X$ssl_port_prefix" = "X999" ]; then
    ssl_port_prefix=$i
  elif [ "X$port_list" = "X999" ]; then
    
    # set ports
    if [ "$i" = "11210" ]; then
      external_port=${external_port_prefix}1207
      proxy2_port=${proxy2_port_prefix}1207
      ssl_port=${ssl_port_prefix}1207
    else
      external_port=${external_port_prefix}${i}
      proxy2_port=${proxy2_port_prefix}${i}
      ssl_port=${ssl_port_prefix}${i}
    fi

    ####################################
    # INCOMING RULES

    # Allow loopback access to intermediate ports so that proxy1 can
    # route traffic to proxy2
    iptables -A chain-couchbase-incoming -i lo -p tcp --dport ${proxy2_port} -j ACCEPT

    # Disallow external access to proxy2
    iptables -A chain-couchbase-incoming -p tcp --dport ${proxy2_port} -j REJECT

    # Disallow direct access to encrypted ports
    iptables -A chain-couchbase-incoming -p tcp --dport ${ssl_port} -j REJECT

    # Allow loopback access to unencrypted ports so that proxy2 can
    # route traffic to DB
    iptables -A chain-couchbase-incoming -i lo -p tcp --dport ${i} -j ACCEPT

    # Disallow direct access to unencrypted ports
    iptables -A chain-couchbase-incoming -p tcp --dport ${i} -j REJECT

    # Allow access to proxy1
    iptables -A chain-couchbase-incoming -p tcp --dport ${external_port} -j ACCEPT

    ####################################
    # OUTGOING RULES

    # Allow loopback access to unencrypted ports to allow routing from proxy2 to
    # DB (proxy2 runs under couchbase UID)
    iptables -A chain-couchbase-outgoing -o lo -p tcp --dport $i -m owner --uid couchbase -j ACCEPT

    # Allow loopback access to intermediate ports to allow routing from proxy1 to
    # proxy2 (proxy1 runs under couchbase UID)
    iptables -A chain-couchbase-outgoing -o lo -p tcp --dport ${proxy2_port} -m owner --uid couchbase -j ACCEPT

    # Disallow loopback access to unencrypted ports to prevent local clients from
    # skipping interception
    iptables -A chain-couchbase-outgoing -o lo -p tcp --dport $i -j REJECT
  fi
done

if [ "${port_list}" = "0" ]; then
     echo “usage: /root/set_firewall.sh -external_port_prefix [1-9] -proxy2_port_prefix[1-9] -ssl_port_prefix [1-9] -port_list [servers’s no SSL port list]”
    echo “for example /root/set_firewall.sh -external_port_prefix 2 -proxy2_port_prefix 3 -ssl_port_prefix 1 -port_list 8091 8092 8093 8094 11210”
fi

# Firewall chains need to return at the end
iptables -A chain-couchbase-incoming -j RETURN
iptables -A chain-couchbase-outgoing -j RETURN

# Hook the main rules up to the chains
iptables -A INPUT  -j chain-couchbase-incoming
iptables -A OUTPUT -j chain-couchbase-outgoing

Configure the S-TAP.
Use the first instance of the proxy to terminate the encryption and add proxy-protocol for Guardium to collect the traffic and be able to attribute the correct analyzed_client_ip. Use the second instance to remove proxy-protocol, in order to not break the connection to the database. Configure the inspection engine to collect the traffic between the two proxy instances. For example, if you are using the sample configurations above, then the ports to collect for Couchbase are 28091-28094, 11210, 21207. If local TCP connections are allowed, then iptables rules need to allow for the possibility that the UID can vary depending on your system configuration. And as with all firewall rules, order is important.
Edit the inspection engine for port_range_end, port_range_start and real_db_port. They should be set as proxy2 port or range and db_type should be COUCHBASE. For example, when proxy2 port is 2xxxx:
- If you are using K-TAP, modify the IE parameters in the guard_tap.ini file:
```
[DB_0]
db_type=COUCHBASE
port_range_end=28094
port_range_start=28091
real_db_port=28091
networks=127.0.0.1/255.255.255.255,9.70.165.199/255.255.255.255
```
- If you are using PCAP, modify the parameters in the guard_tap.ini file:
```
devices=ens32,lo
ktap_installed=0

[DB_0]
db_type=COUCHBASE
port_range_end=28094
port_range_start=28091
real_db_port=28091
```