Terracotta Cluster Tests

Edit online

All tests in this section should be run after the Network Tests succeed.

Test Plan - Active L2 System Loss Tests - verify Mirror Takeover

The test plan for mirror takeover consists of the following tests:

TestID Test Setup Steps Expected Result

TAL1

Active L2 Loss - Kill

L2-A is active, L2-B is mirror. All systems are running and available to take traffic.

  1. Run app
  2. Kill -9 Terracotta PID on L2-A (Active)

L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

TAL2

Active L2 Loss - clean shutdown

L2-A is active, L2-B is mirror. All systems are running and available to take traffic.

  1. Run app
  2. Run ~/bin/stop-tc-server.sh on L2-A (Active)

L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

TAL3

Active L2 Loss - Power Down

L2-A is Active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Power down L2-A (Active)

L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

TAL4

Active L2 Loss - Reboot

L2-A is Active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Reboot L2-A (Active)

L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

TAL5

Active L2 Loss - Pull Plug

L2-A is Active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Pull the power cable on L2-A (Active)

L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

Test Plan - Mirror L2 System Loss Tests

System loss tests confirms High Availability in the event of loss of a single system. This section outlines tests for testing failure of the Terracotta mirror server.

The test plan for testing Terracotta mirror Failures consist of the following tests:

TestID Test Setup Steps Expected Result

TPL1

Mirror L2 loss - kill

L2-A is active, L2-B is mirror. All systems are running and available to take traffic.

  1. Run app
  2. Kill -9 L2-B (mirror)

data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

TPL2

Mirror L2 loss -clean

L2-A is active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Run ~/bin/stop-tc-server.sh on L2-B (mirror)

data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

TPL3

Mirror L2 loss -power down

L2-A is active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Power down L2-B (mirror)

data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

TPL4

Mirror L2 loss -reboot

L2-A is active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Reboot L2-B (mirror)

data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

TPL5

Mirror L2 loss -Pull Plug

L2-A is active, L2-B is mirror. All systems are running and available to take traffic

  1. Run app
  2. Pull plug on L2-B (mirror)

data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

Test Plan - Failover/Failback Tests

This section outlines tests to confirm the cluster ability to fail-over to the mirror Terracotta server, and fail back.

The test plan for testing fail over and fail back consists of the following tests:

TestID Test Setup Steps Expected Result

TFO1

Failover/Failback

L2-A is active, L2-B is mirror. All systems are running and available to take traffic

  1. Run application
  2. Kill -9 (or run stop-tc-server) on L2-A (Active)
  3. After L2-B takes over as Active, start-tc-server on L2-A. (L2-A is now mirror)
  4. Kill -9 (or run stop-tc-server) on L2-B. (L2-A is now Active)

After first failover L2-A->L2-B, txns should continue. L2-A should come up cleanly in mirror mode when tc-server is run. When second failover occurs L2-B->L2-A, L2-A should process txns.

Test Plan - Loss of Switch Tests

Tip: This test can only be run on a redundant network

This section outlines testing the loss of a switch in a redundant network, and confirming that no interrupt of service occurs.

The test plan for testing failure of a single switch consists of the following tests:

TestID Test Setup Steps Expected Result

TSL1

Loss of 1 Switch

2 Switches in redundant configuration. L2-A is active, L2-B is mirror. All systems are running and available to take traffic.

  1. Run application
  2. Power down/pull plug on Switch

All traffic transparently moves to switch 2 with no interruptions

Test Plan - Loss of Network Connectivity

This section outlines testing the loss of network connectivity.

The test plan for testing failure of the network consists of the following tests:

TestID Test Setup Steps Expected Result

TNL1

Loss of NIC wiring (Active)

L2-A is active, L2-B is mirror. All systems are runnng and available to traffic

  1. Run application
  2. Remove Network Cable on L2-A

All traffic transparently moves to L2-B with no interruptions

TNL2

Loss of NIC wiring (mirror)

L2-A is active, L2-B is mirror. All systems are runnng and available to traffic

  1. Run application
  2. Remove Network Cable on L2-B

No user impact on cluster

Test Plan - Terracotta Cluster Failure

This section outlines the tests to confirm successful continued operations in the face Terracotta Cluster failures.

The test plan for testing Terracotta Cluster failures consists of the following tests:

TestID Test Setup Steps Expected Result

TF1

Process Failure Recovery

L2-A is active, L2-B is mirror. All systems are running and available to traffic

  1. Run application
  2. Bring down all L1s and L2s
  3. Start L2s then L1s

Cluster should come up and begin taking txns again

TF2

Server Failure Recovery

L2-A is active, L2-B is mirror. All systems are running and available to traffic

  1. Run application
  2. Power down all machines
  3. Start L2s and then L1s

Should be able to run application once all servers are up.

Client Failure Tests

This section outlines tests to confirm successful continued operations in the face of Terracotta client failures.

The test plan for testing Terracotta Client failures consists of the following tests:

TestID Test Setup Steps Expected Result

TCF1

L1 Failure -

L2-A is active, L2-B is mirror. 2 L1s L1-A and L1-B All systems are running and available to traffic

  1. Run application
  2. kill -9 L1-A.

L1-B should take all incoming traffic. Some timeouts may occur due to txns in process when L1 fails over.