Swift PUT returns the 202 error and S3 PUT returns the 500 error due to the missing time synchronization

Description

The swift object servers require monotonically-increasing timestamps on the PUT requests. If the time between all the nodes is not synchronized, the PUT request can be rejected, resulting in the object server returning a 409 status code that is turned into 202 in the proxy-server. When the swift3 middleware receives the 202 code, it returns a 500 to the client. When enabling DEBUG logging, the system displays the following message:

From the object server:

Feb 9 14:41:09 prt001st001 object-server: 10.0.5.6 - - [09/Feb/2016:21:41:09 +0000] "PUT /z1device119/14886/AUTH_bfd953e691c4481d8fa0249173870a56/mycontainers12/myobjects407"

From the proxy server:

Feb 9 14:14:10 prt003st001 proxy-server: Object PUT returning 202 for 409: 1455052450.83619 <= '409 (1455052458.12105)' (txn: txf7611c330872416aabcc1-0056ba56a2) (client_ip:

If S3 is used, the following error is displayed from Swift3:

Feb 9 14:25:52 prt005st001 proxy-server: 500 Internal Server Error: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 81, in __call__#012 resp = self.handle_request(req)#012 File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 104, in handle_request#012 res = getattr(controller, req.method)(req)#012 File "/usr/lib/python2.7/site-packages/swift3/controllers/obj.py", line 97, in PUT#012 resp = req.get_response(self.app)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 825, in get_response#012 headers, body, query)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 805, in get_acl_response#012 app, method, container, obj, headers, body, query)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 669, in _get_response#012 raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: tx40d4ff7ca5b94b1bb6881-0056ba5960) (client_ip: 10.0.5.1) Feb 9 14:25:52 prt005st001 proxy-server: 500 Internal Server Error: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 81, in __call__#012 resp = self.handle_request(req)#012 File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 104, in handle_request#012 res = getattr(controller, req.method)(req)#012 File "/usr/lib/python2.7/site-packages/swift3/controllers/obj.py", line 97, in PUT#012 resp = req.get_response(self.app)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 825, in get_response#012 headers, body, query)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 805, in get_acl_response#012 app, method, container, obj, headers, body, query)#012 File "/usr/lib/python2.7/site-packages/swift3/request.py", line 669, in _get_response#012 raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: tx40d4ff7ca5b94b1bb6881-0056ba5960) (client_ip: 10.0.5.1)

Cause

The system displays these errors when the time is not in sync.

Proposed workaround