IBM Support

IJ05649: 'DEPLOY CHANGES' CAN SOMETIMES CAUSE A DROP IN CONNECTION BETWEEN ECS-EC AND ECS-EP LEADING TO EVENTS BEING DROPPED

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • It has been identified that after a QRadar 'Deploy Changes'
    function is performed (manual or automated triggered by
    AutoUpdate), connection timeout issues between the ecs-ec and
    ecs-ep services can occur in some instances.
    When this connection timeout issue occurs, event collection
    continues but event processing functions (including writing to
    disk) do not occur, causing events to be dropped.
    
    Messages similar to the following might be visible in
    /var/log/qradar.log on affected QRadar appliances when this
    issue is occuring:
    
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]
    com.q1labs.semsources.destinations.StoreForwardDestination(ecs-e
    c/EC/TCP_TO_EP): [WARN] [NOT:0000004000][127.0.0.1/- -] [-/-
    -]Socket Write Timeout
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]
    java.net.SocketTimeoutException: Write socket timeout 60000 ms
    expired
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.frameworks.nio.network.protocol.UnencryptedProtocolIm
    p.writeBufferToChannelInternal(UnencryptedProtocolImp.java:96)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.frameworks.nio.network.protocol.Protocol.writeBufferT
    oChannelInternal(Protocol.java:767)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.frameworks.nio.network.protocol.Protocol.writeToChann
    el(Protocol.java:789)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.frameworks.nio.network.CommunicatorBase.writeToChanne
    l(CommunicatorBase.java:279)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.frameworks.nio.network.Communicator.writeToChannel(Co
    mmunicator.java:287)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.semsources.destinations.StoreForwardDestination$TakeF
    romQueueJob.sendEventFromQtoCommunicator(StoreForwardDestination
    .java:156)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.semsources.destinations.StoreForwardDestination$TakeF
    romQueueJob.run_internal(StoreForwardDestination.java:199)
    [ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at
    com.q1labs.semsources.destinations.StoreForwardDestination$TakeF
    romQueueJob.run(StoreForwardDestination.java:176)
    
    [ecs-ep.ecs-ep] [ReceiverServer:ecs-ep/EP/Q1From_EC_via_TCPIP]
    com.q1labs.semsources.sources.TCPSource(ecs-ep/EP/Q1From_EC_via_
    TCPIP): [WARN] [NOT:0000004000][127.0.0.1/- -] [-/- -]Queue
    timeout (10000 milliseconds) occurred. Dropping event
    
    [ecs-ep.ecs-ep] [ReceiverServer:ecs-ep/EP/Q1From_EC_via_TCPIP]
    com.q1labs.semsources.sources.TCPSource(ecs-ep/EP/Q1From_EC_via_
    TCPIP): [INFO] [NOT:0000006000][127.0.0.1/- -] [-/- -]Following
    message suppressed 30 times in 300000 milliseconds
    [ecs-ep.ecs-ep] [ReceiverServer:ecs-ep/EP/Q1From_EC_via_TCPIP]
    com.q1labs.semsources.sources.TCPSource(ecs-ep/EP/Q1From_EC_via_
    TCPIP): [WARN] [NOT:0000004000][127.0.0.1/- -] [-/- -]Queue
    timeout (10000 milliseconds) occurred. Dropping event
    

Local fix

  • A manual restart of the ecs-ep service on affected QRadar
    appliance(s) can sometimes correct this behavior.
    
    # systemctl restart ecs-ep
    
    The issue has been seen to occur following an AutoUpdate of
    geodata and the subsequent Deploy function that accompanies the
    update.  That geodata content update can be disabled by going
    into Admin -> System Settings -> Geographic Settings and set
    "Disable Automatic Content Updates" to "True".
    

Problem summary

  • This issue was fixed in QRadar QRM QVM release of 731 Patch 5
    interimfix 01 and 731 Patch 6.
    

Problem conclusion

  • This issue was fixed in QRadar QRM QVM release of 731 Patch 5
    interimfix 01 and 731 Patch 6.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IJ05649

  • Reported component name

    QRADAR SOFTWARE

  • Reported component ID

    5725QRDSW

  • Reported release

    731

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2018-04-11

  • Closed date

    2018-12-11

  • Last modified date

    2018-12-11

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    QRADAR SOFTWARE

  • Fixed component ID

    5725QRDSW

Applicable component levels

[{"Business Unit":{"code":"BU048","label":"IBM Software"}, "Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"731","Edition":""}]

Document Information

Modified date:
11 December 2018