Stopping a multithreaded replication supplier

Edit online

In a replication environment, abruptly stopping a supplier that uses multithreaded replication to accelerate replication between its consumers can cause problems. To avoid having replication-related errors, complete the specified steps before you stop a supplier server.

About this task

At any specified time, a supplier might send multiple updates to a consumer. , The number of updates can be the number of consumer connections, which are multiplied by the depth of replication receive queue. A supplier can also have multiple consumers. In such cases, the number of replication updates at any specified time can be large. The replication status of the supplier is based on the most recent change replicated (successfully or otherwise) so far. If a supplier is restarted, it uses this replication status value to determine the changes that must be sent to the consumer. If a supplier is stopped abruptly before it receives a response from its consumers for the updates that it sent, the supplier sends the updates again. When these updates are sent again, it can cause errors to be reported. These errors are logged by the supplier. They can be cleared by using the Web Administration Tool or the command-line utility for managing the replication error log. If the logging of replication errors is enabled, these errors are counted against the limit for logged errors. Hence, they must be cleared.

If the logging of replication errors is not enabled, these kinds of errors that occur do not block replication. The replicated add operations report that the entries exist. The modify operation reports that the attribute values exist or are not found. The delete operations report that the entries no longer exist. The consumer server might log these errors but replication continues.

In some cases, the administrator might not be aware of the problem and hence might not be able to resolve the problem. A replicated update from a supplier to its consumer might result in an error and the supplier might not be available to log the error. Depending on the last response that is received by the supplier, this update might not be replicated again.

Procedure

Find the server ID of a supplier. An example of the ldapsearch command with its output:

#ldapsearch -D cn=admin -w password -p 2389 -s base objectclass=* ibm-serverID
ibm-serverId=wingspread-2389

Find all the replication subentries with the server ID obtained in step 1.

#ldapsearch -D cn=admin -w password -p 2389 -s sub \
   ibm-replicaServerId=wingspread-2389 0.0

ibm-replicaServerId=wingspread-2389,ibm-replicaGroup=default,O=SAMPLE

Find all the replication agreements for the server.

#ldapsearch -D cn=admin -w password -p 2389 -b ibm-replicaServerId=wingspread-2389,\
   ibm-replicaGroup=default,O=SAMPLE objectclass=ibm-replicationAgreement 0.0

cn=wingspread-1389,ibm-replicaServerId=wingspread-2389,ibm-replicaGroup=default, 
O=SAMPLE

Verify the status for a specified replication agreement.

#ldapsearch -D cn=admin -w password -p 2389 -b cn=wingspread-1389,\
   ibm-replicaServerId=wingspread-2389,ibm-replicaGroup=default,O=SAMPLE \
   objectclass=* +ibmrepl

cn=wingspread-1389,ibm-replicaServerId=wingspread-2389, 
   ibm-replicaGroup=default,O=SAMPLE ibm-replicationChangeLDIF=N/A 
ibm-replicationLastActivationTime=20080707152436Z 
ibm-replicationLastChangeId=4855 ibm-replicationLastFinishTime=20080707152436Z 
ibm-replicationLastResult=N/A ibm-replicationLastResultAdditional=N/A 
ibm-replicationNextTime=N/A ibm-replicationPendingChangeCount=0 
ibm-replicationState=ready ibm-replicationFailedChangeCount=0 
ibm-replicationperformance= 
   [c=0,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1] 
ibm-replicationperformance= 
   [c=1,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1] 
ibm-replicationperformance= 
   [c=2,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1] 
ibm-replicationperformance= 
   [c=3,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1]

The +ibmrepl in the search filter returns operational attributes that are related to replication. The attribute names are on the left of the equal signs. In the example, there are four connections to the consumer. Some replication status information attributes are only used with single threaded replication, (displayed with the value N/A), others are only for multiple threaded replication. Use ++ibmrepl to show all the attributes, including the pending changes and logged replication errors.

Suspend replication for the agreement.

# ldapexop -D cn=admin -w password -p 2389 -op controlrepl -action suspend \ 
  -ra cn=wingspread-1389,ibm-replicaServerId=wingspread-2389,\  
  ibm-replicaGroup=default,O=SAMPLE

Operation completed successfully.

Verify the status of replication agreement.

#ldapsearch -D cn=admin -w password -p 2389 -b cn=wingspread-1389,\
   ibm-replicaServerId=wingspread-2389,ibm-replicaGroup=default,O=SAMPLE \
   objectclass=* ++ibmrepl

cn=wingspread-1389,ibm-replicaServerId=wingspread-2389,
   ibm-replicaGroup=default,O=SAMPLE
ibm-replicationChangeLDIF=N/A 
ibm-replicationLastActivationTime=20080707152648Z 
ibm-replicationLastChangeId=4855 
ibm-replicationLastFinishTime=20080707152648Z 
ibm-replicationLastResult=N/A 
ibm-replicationLastResultAdditional=N/A 
ibm-replicationNextTime=N/A 
ibm-replicationPendingChangeCount=1 
ibm-replicationState=on hold 
ibm-replicationFailedChangeCount=0 
ibm-replicationperformance=
   [c=0,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1]
ibm-replicationperformance=
   [c=1,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1]
ibm-replicationperformance=
   [c=2,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1]
ibm-replicationperformance=
   [c=3,l=10,op=0,q=0,d=0,ws=0,s=0,ds=0,wd=0,wr=0,r=0,e=0,ss=1,rs=1]
ibm-replicationPendingChanges=4856 modify CN=WINGSPREAD-1389,
   IBM-REPLICASERVERID=wingspread-2389,IBM-REPLICAGROUP=DEFAULT,O=SAMPLE

The pending change that is reported in the output was caused by the operation to suspend replication.

Determine whether there are any replicated updates that were sent to the consumer. In the output from step 6 related to the replication status, the attribute ibm-replicationperformancecan be used to determine the number of updates that were sent to the consumer. This attribute applies only to replication agreements by using multithreaded replications.
The information about the data that is associated with the ibm-replicationperformance attribute in the example output of step 6 is as follows:

c

Indicates the connection number. In the output of step 6, there are four connections. The first connection shows the most traffic. The workload determines how often the other connections are used.

l

Indicates the size limit for each queue. In the example, the value is 10. For each connection, there are two queues of same length. One queue is for updates to be sent on the connection, which is called the send queue. The other queue is for updates that were sent but no response was received from the consumer, which is called the receive queue. When updates are sent, they are moved from send queue to the receive queue. When the receive queue reaches its size limit, no more updates are sent until some responses from the consumer are received. When the send queue reaches its size limit, no more updates are assigned to the connection. When the size limit for all the send queues of connections are reached, the supplier waits for the consumer to process the backlog.

op

Specifies the replication ID of the last operation that is assigned to the send queue of the connection. Replication IDs are assigned to all updates that are to be replicated to a consumer. The process of assigning replication IDs must not stop even if replication is suspended.

q

Specifies the current size of the send queue. This value must not change if replication is suspended.

d

Specifies the count of dependent updates. An add request for an entry followed by a modify request of the same entry is counted as a dependency. All dependent updates must be assigned to the same connection so that they can be applied in correct order.

ws

Indicates the number of times the send queue reached its size limit.

ds

Specifies the number of dependent updates sent.

wd

Specifies the number of times that the send queue waited for a dependent update before it sent more updates.

wr

Indicates the number of times the receive queue reached its size limit.

r

Indicates the number of replicated updates that is waiting for a response from the consumer.

e

Specifies the number of replication errors reported by the consumer.

ss

Indicates the session count of the sender thread. It is incremented when a connection to the consumer is established.

rs

Indicates the session count of the receiver thread.

The value indicated by r might show that the number of replicated updates that are waiting for a response is 0. In this case, it is safe to stop the supplier for this consumer server. The value of r varies between 0 and the value of l, the size limit of the queue, which defaults to 10. If the value is not 0 for r, you must wait for it to be 0. The value of r depends on the size and type of the replication update and the workload on the consumer. After this value becomes 0, the supplier sends the status of updates to the consumer. On restarting the supplier, it replicates only the updates that were not sent before.
Repeat the steps 4 through 7 for each replication agreement that is serviced by the supplier.
Stop the supplier server when the replicated update status is 0 for all the consumers.

After you restart the supplier, resume the replication that was suspended in step 5.

#ldapexop -p 2389-D cn=admin -w password -op controlrepl -action resume \ 
-ra cn=wingspread-1389,ibm-replicaServerId=wingspread-2389,\ 
ibm-replicaGroup=default,O=SAMPLE
Operation completed successfully.

Results

You can also use Web Administration Tool to see the status of a replication agreement and suspend or resume replication. Use a similar approach to determine when there are not any replicated updates whose status is not reflected on the supplier.