One of the characteristics of ER  is that it uses deferred constraint checking.  This means that constraints are checked as part of the commit and not as part of the update operation.  Defered constraint checking has a huge advantage for ER because that means we can dynamically increase the parallelism by the apply while still supporting constraints such as referential integrety.  For instance, this allows us to apply the transaction which  might create a parent row and a seperate transaction which creates a child row at the same time.  All we have to do is to ensure that 'parent transaction' commits prior to the 'child transaction'.  Since ER uses deferred constraint checking, by coordinating the order of commits, we can make sure that the parent row will exist prior to the deferred constraint checking being performed by the child transaction.  This significantly improves the apply of ER as it allows the apply to take maximum advantage of the resources which are available on the target node.  The transactions on the source were goverened by the application.  The application controlled the order of the activities on the source node.  That would mean that only the resource usage was limited to what the application could take advantage of.  Usually this will mean serialized operations.  However, the target has no such restriction and thus can take advantage of all resources.  That's why generally ER can catch up after an outage in a much shorter time than the origional activity took.


Let's see how this works.  Figure 1 shows what will generally occur on a source node when a session performs two transactions in which the first will insert a parent row into one table and then commit.  The session then will open a subsequent transaction in which it will insert a child row into another table in which there is a relationship with the parent table (referential integrety).  This requires that two serialized transactions be executed.

On the other hand, figure 2 describes what will happen on the target node.  ER does not know that the original transactin was done by a single session within two transactions, nor does it care.  The goal is to apply the operations on the target as quickly as possible.  That means that the ER apply transactions must take advantage of all of the resources available.  So what we do is to sense that there is a referential integrety relationship between the parent and child tables and then to guarentee that the parent transaction will commit prior to the child transaction.  Since ER is using deferred constraint checking, that means that the constraint rules are checked during the commit.  Therefore, the apply is able to overlay the transactions and only serialize the commits.

It is generally best that systems using ER use constraints in general - especially if there are unique indexes besides the primary key.  By making those unique indexes into unique constraints, ER is better able to ensure that the transaction will be successfully applied when the unique columns are being updated.  In recent versions of ER, we have added startup warnings if we detect that a unique index exists which is not part of a unique constraint.  

In the novel, "Dune", there is a phrase which is constantly repeated - "The spice must flow".  Well - a similar thing must happen with HDR, except with HDR it's "The Logs Must Flow".   Let me explain by examining the following diagram.

HDR works by transferring the logs to the secondary where the recovery component applys those logs.  The secondary is in perpetual recovery mode.  

The logs are transfered to the secondary by copying the log buffer into an HDR transmit buffer as part of the flush of the log buffer to disk.  If using synchronous mode of HDR, then the HDR Transmit buffer is immediately scheduled for transmission and the thread which caused the log flush is held until the ACK of that transmission is received from the secondary.

If using asynchronous HDR, then the HDR transmit buffer is not scheduled for transmission until either the transmit buffer is full or until it has aged to the DRINTERVAL time limit.

While the HDR transmit buffer is sized the same size as the log buffer, it is not a 1 to 1 relationship.  If a log buffer is flushed with only part of the buffer being filled (as is often the case with unbuffered committed transactions), then only part of the HDR transmit buffer is used.  When the next log flush occurs, we could then copy the new log pages into  the remainder of that HDR transmit buffer.

The transmission of the HDR transmit buffer is sent to the secondary using a half-duplex protocol.  That means that we can not send the next buffer until we receive an ACK for the previous transmission.  While this may increase the delay in sending a log buffer, it also is necessary to ensure one of the main characteristics of HDR.  That is the assurance that the secondary can take on the role of the primary with no loss of committed transactions.
In order to receive the HDR transmit buffer, the HDR receive thread must first obtain a buffer from the HDR receive buffer pool.  If it can not obtain a buffer, it will wait until the recovery threads place and empty buffer into the pool.  When the receive thread receives a buffer from the primary, it will ACK the buffer and then queue the received buffer of log pages to the recovery component.  While it is a bit more complex, for our purposes we will think of the recovery threads consisting of a main recovery thread an a bunch of worker recovery threads.  

The main recovery thread will split the log page into log records and then queue that log record to one of the worker recovery threads based on the partition number of that log record.  All of the log records for a given partition will be applied serially by the same worker recovery thread.  Some of the log records are processed by the main recovery thread only after all of the worker threads have processed the log records queued to them.  These log records can be considered as a globally serialized log record.  The checkpoint log record is one such log record.  

After the main recovery component is finished with an HDR buffer, that buffer is placed in the receive HDR buffer pool.

From this we can see that if we don't process things efficiently on the secondary, then we will not be able to return an HDR receive buffer into the receive queue quickly.  If we can't get the receive buffer into the HDR receive buffer pool quickly, then we will  not be able to easily get an HDR receive buffer in which to receive the data transmission.  If we don't receive a transmission quickly, then we can't send the next buffer from the source due to half-duplex transmission.  If we can't send a buffer from the source, then we can't return that full HDR transmit buffer to the HDR transmit buffer pool.  If we can't return the HDR transmit buffer quickly to the HDR transmit buffer pool quickly, then the log flush logic is unable to get an empty HDR transmit buffer easily.  If the log flush logic is not able to easily get an empty HDR transmit buffer, then logging will be blocked.  A problem with the apply on the secondary can back flow and impact the primary. So as Mr. Herbert said, "The spice must flow".

Indexed Spices

There is an additional consideration which needs to be considered and that's what happens when an index is created.  At the end of the index creation, we transfer the index to the secondary server.  This is done by having the thread which created the index to copy the newly created index into an HDR transfer buffer and sending the index to the secondary.  Because of the increased usage of the transmission buffers by the index transfer, there can be some degredation in the log transfer when an index is created.  One of the things that we did with IDS 11 was to implement a feature called Index Page Logging.  This allows the transfer of the log pages to be done by placing the index pages into the log itself.  To avoid a long transaction, the index page logging is actually done within multiple transactions.  

While this does increase the log consumption, it also ensures that the index transfer does not use all of the transfer buffers and allows non-index log pages to be intermixed with the index log pages.  This tends to equalize the impact of the index transfer with the user esql threads.

Half 'n Half

As was mentioned earlier, the transmission of HDR buffers to the secondary uses a half-duplex protocol.  This is absolutly critical in order to support failover with no loss of committed transactions, but is also critical if we want to do a 'flip-flop'  That is the case when the secondary becomes the primary and the primary becomes the secondary.  With IDS11, we implemented the RSS secondary which does not use half-duplex protocol, but instead uses full duplex with flow control.  This eliminates the impact of half-duplex protocol.  Additionally the RSS node does not block the log flush threads.  Because of this, the RSS node can be considered when willing to use asynchronized mode with HDR.