Topic
  • No replies
SystemAdmin
SystemAdmin
367 Posts

Pinned topic Troubleshooting - flow collection, netflow collection, external flow collection

‏2011-07-28T14:18:57Z |
Many users have called in with problems around collecting flow data, both packet based sources and external/netflow sources. This post will try to cover the most common issues with this topic.

- Flow collection processes
QRadar uses a component called "qflow" to perform flow collection. This component can be seen in the QRadar Deployment Editor (versions 5.x -> 7.0). In 6.3.1, qflow woudl send data to the "flowprocessor" component, which would then feed up to 1 or more "classify" processes. In 7.0, the "flowprocessor" and "classify" processes have been removed, as has the "flow view" tab in the deployment editor, and now flow information passes through the "event collector" pipeline.

A common issue after upgrading from 6.3.1 to 7.0.0, is that the qflow process is not automatically connected to the eventcollector component during the upgrade. If you have just upgraded to 7.0 and are not seeing flows, make sure you check this connection in the deployment editor.

Another common issue is that while a qflow is configured in the deployment editor, it may NOT have any flow sources assigned to it in the "Flow Sources" window in the admin tab. If this is the case, you may see dashboard notifications around the qflow process failing to start multiple times. If you see this, check the flow sources window to ensure there are sources assigned to that qflow.
- Packet based flow collection
This collection method relies on the use of span & mirror ports on switches, or inline network taps. Depending on method in use, you may use 1 or 2 ports on your flow collector to capture this data. When using network ports, ensure that the ports in use (typically eth1 to eth3) are configured in "Flow Sources", and are assigned to the qflow in question.

In 1201/1302 collectors, you can also SSH in to your collector and use linux commands to verify data. If you run the command "ifconfig -a", you can check the interface list. Interfaces marked with "RUNNING" have a link, while interfaces without that do not have a link. You can also use the tool "ethtool eth[n]" to verify speed & duplex of the interface.

Verifying the data itself, you can use the command "tcpdump -i eth[n]" to view the data on that port.

When using 1202/1301/1310 collectors, these systems use dedicated high speed cards for data collection. These cards do not work with tcpdump, but other tools are available. See the qmmunity post at https://qmmunity.q1labs.com/node/1180 for more details.

- external (netflow/sflow) collection
Again, similar to external packet collection, ensure you have the proper external flow source configured in the "Flow Sources" section of the admin tab. Without those configured, you will not receive external flow data.

Once you've verified the sources are assigned, you may want to check the collector to ensure the netflow packets are coming in. Here is a quick walk through to test this:

  • ssh to the collector which the flow source is configured for.
  • check the interface with "tcpdump" to ensure data is arriving on the interface and port specififed
tcpdump -nnAs0 -i eth0 port [N]
If no data is visible, ensure your router is sending to the proper destination address and that there are no firewalls between your router & QRadar flow collector.

Once you verify that data is present, you may want to take a look inside the data to ensure proper records are being sent. To do this, you can capture a sample, download it, and open it up in wireshark. This is a great method to ensure that the data you are getting is complete - often, external flow data, will be reported with only source or destination bytes, and be considered "unidirectional" by QRadar and can cause rules looking for that kind of data to fire & generate offenses.
tcpdump -nnAs0 -c 500 -w netflow.sample.500packets-port2055.pcap -i eth0 port 2055
Once captured, scp the file to a Windows box with Wireshark. Open in Wireshark and you should see Cisco Netflow as an expandable option per packet. This should contain the flow information

You can also use a utility written by Q1 Labs to monitor data as it comes in off the wire. This utility, "efDump" runs similar to tcpdump, however it has the ability to reach into the netflow datagram packets, and print out the netflow records themselves. This can be quite useful for a quick check of the data coming into your system.

Also note, some of these specific posts around some devices that have seen repeated configuration issues:

________________________________
dwight s.

-------Posted BY dwight (q1)