Network Traffic Guide

3 likes Updated 5/9/16, 4:09 PM by Lee WeiTags: None

BigFix is designed to be a real time (seconds and minutes) system capable of working in extremely limited bandwidth environments. Properly configured, The BigFix platform can effectively and quickly deploy files that are hundreds of MBs large to tens of thousands of computers without significantly affecting general network performance. Because of the speed and prevalence of the BigFix platform in the network, however, it is very important that network engineers understand the different types of traffic that the BigFix Servers, Relays, Clients, and Consoles can generate.

The following information describes the different types of network traffic and explains the purpose of the traffic as well as information about how to handle "Possible Worst Case Traffic".

Note: The "possible worst case traffic" are definitely not expected; however, in the interest of preparing for the worst case scenario, example situations are given as well mechanisms to recover from the situation. This information is intended to educate network engineers and provide the basis of a disaster recovery procedure related to the BigFix infrastructure.

Note: All port numbers provided within this document describe the destination port, not source port of generated network traffic except where explicitly stated.  When configuring firewalls within your environment, please assume source ports will be random and perform any port-based filtering based upon only the destination port.

Refer to the following diagram in the network discussions below:

 

 

A

           
  Network Traffic:       Gather / Post / Download
           
  Purpose:       The BigFix Client will contact its BigFix Relay (or BigFix Server) for several reasons including: gathering the latest information about actions and Fixlet messages, sending results about the computer properties or status of actions/Fixlets, and downloading files (such as patches).
           
  Port:       52311 (configurable by the BigFix Administrator at install time)
           
  Protocol:       TCP (HTTP)
           
  Origination:       The BigFix Client initiates the request to the BigFix Relay (or BigFix Server).
           
  Expected Traffic Frequency:        
  • Gather: The BigFix Clients will gather the latest Fixlet content or actions once a day OR whenever a new action or new Fixlets are available from the BigFix Relay. Each gather is approximately 1 KB - 3 KB (only compressed differences are gathered). An active deployment will have this occur many times (up to many hundreds of times a day when the BigFix Console operators are very active)
  • Post: The BigFix Clients will send results of their actions / Fixlets (approximately 1 KB each submission) for each action. The BigFix Clients will also send "heartbeats" periodically (configurable by BigFix Administrator - default 15 minutes) to the BigFix Relay of about 1 KB that contain the differences in the BigFix Clients properties.
  • Downloads: Downloads occur only when a BigFix Console operator sends out an action to tell a BigFix Client to download and run a file. The frequency of this operation is completely dependent on the activity of the BigFix Console Operators.

Note: If the BigFix infrastructure is properly configured, the traffic described is between the BigFix Client and the BigFix Relay (which is usually a high bandwidth connection) and the information between the BigFix Relay and the BigFix Server (sometimes a slow bandwidth connection) is minimal: gather/download information sent only once and posts are compressed and forwarded to the BigFix Server.

           
  Expected Traffic Volume:        
  • Gather: Low - Perhaps 5 KB - 50 KB per day per BigFix Client spread throughout the day.
  • Post: Low - Perhaps 10 KB - 50 KB per day per BigFix Client spread throughout the day.
  • Downloads: High - If no actions are pushed, then no downloads will occur. If a patch is downloaded, the BigFix Client will download the file (30 KB - 300 MB) from the BigFix Relay.

Note: This traffic varies heavily depending on usage. A very active deployment (BigFix Console operators sending lots of actions) might see numbers that are 10x the numbers above.

           
  Possible Worst Case Traffic:      

In a properly configured BigFix deployment, the BigFix Relays always have a high-bandwidth connection to the BigFix Relays to minimize network traffic on the slow network connections. The worst case scenario is if the BigFix Relays are improperly configured or the if the BigFix Clients or BigFix Relays fail in some form while a large download is being pushed. For instance, if all the BigFix Relays simultaneously failed while Windows XP SP2 (250 MB) action is sent, the BigFix Clients will all begin to download the file from the BigFix Server. In such a scenario, all the slow network connections between the BigFix Clients and BigFix Server will be flooded with HTTP traffic on port 52311.

This situation can be recovered from by stopping the action immediately in the BigFix Console (depending on how busy the network is, this could take 2 mins - 2 hours), blocking TCP on port 52311 at the network routers, or by shutting down the BigFix Server/BigFix Relays (this can be done through BigFix, thus eliminating the source of the download servers).

           
  Network Controls:       The BigFix platform provides the following controls for this type of traffic:

 
  • Configurable bandwidth throttling to BigFix Relay or BigFix Clients
  • Configurable gather interval (default 1 per day per Fixlet site)
  • Configurable minimum time to wait between posts (default 15 sec)
  • Configurable temporal distribution (spread out downloads over time) per action
  • Ability to set "policy" to prevent computers from downloading files if they are not pointed at the proper BigFix Relay
           
  Other Notes:       Traffic from server to relay is required for force refreshes and for Wake-On-LAN.



One of the major focuses of deploying the BigFix infrastructure is to get the BigFix Relays in locations that are near the BigFix Clients so that WAN traffic is minimized and the vast majority of the network traffic is between the BigFix Clients and the BigFix Relays on a fast LAN connection. The BigFix platform offers a number of reports to see the status of the BigFix Clients and which BigFix Relays they are pointing to and it is the job of the BigFix Administrators to make sure the BigFix Relay and BigFix Clients are properly configured.
           

 

B

           
  Network Traffic:       UDP "new information" message
           
  Purpose:       When there is new information available such as new Fixlet messages or new actions, a UDP message is sent to the BigFix Clients' last known IP addresses to tell them to gather the lastest data from the BigFix Server.
           
  Port:       52311 (configurable by the BigFix Administrator at install time) [destination port]
           
  Protocol:       UDP
           
  Origination:       The UDP messages are sent from the BigFix Clients' immediate "parent", which can be either a BigFix Relay or the BigFix Server.
           
  Expected Traffic Volume:       Low - The UDP messages are sent from the BigFix Relay to the BigFix Client every time a new action is sent or a new Fixlet message is gathered. The BigFix Relay will send 3 UDP messages to each BigFix Client (to help ensure delivery). Each UDP message is about 60 bytes.
           
  Expected Traffic Frequency:       In an idle deployment (no actions being sent) there will be virtually no UDP traffic. In a very active deployment where many actions are sent, there will be 3 UDP messages sent per action per BigFix Client (as a reference point, sending 100 actions over the course of a day is considered very high BigFix usage). Note that if the BigFix Relays "nearby" the BigFix Clients, the UDP traffic will be sent on the LAN and the WAN will have very little UDP traffic.
           
  Possible Worst Case Traffic:       The worst case scenario for this type of traffic is for the BigFix Relays to fail and send much more traffic than they are supposed. In such a scenario, the BigFix Relays would be repeatedly sending UDP messages to each of the BigFix Clients. There is a control built in for the BigFix Relays never to send more than 500 UDP messages per second (configurable), but if there were many BigFix Relays or many BigFix Clients, this UDP traffic could potentially cause lots of network traffic or router usage. In such a situation, you would see massive amounts of UDP traffic on port 52311 from the BigFix Relays.

To recover from this situation, UDP traffic on port 52311 could be blocked at the routers or you could shut off the BigFix Relays (can be done through BigFix).

           
  Network Controls:       BigFix provides the following controls in the product for this type of traffic:

 
  • Configurable limit of the number of UDP messages sent at one time from a BigFix Relay
  • Configurable limit of the amount of time to wait after sending UDP messages from a BigFix Relay
           
  Other Notes:       It is not strictly necessary for BigFix to use UDP traffic for the system to work properly, but the UDP messages allow the BigFix Clients to be notified of new information as opposed to repeated polling by the BigFix Clients. Thus, the UDP messages can help reduce overall network usage and increase response time.



UDP messages from the BigFix Relay to the Wake-On-LAN forwarding computer are required for Wake-On-LAN to work properly.
           

 

C

           
  Network Traffic:       BigFix Relay Selection
           
  Purpose:       In a well configured BigFix network, each BigFix Client is configured to communicate with a local BigFix Relay; however, manually selecting the BigFix Relay for each BigFix Client can take a lot of time and is error prone. To solve this problem, the BigFix Clients can be configured to find their closest BigFix Relay based on the number of network hops. To find their closest BigFix Relay, the BigFix Client use the ICMP protocol (similar to tracert).
           
  Port:       N/A (the ICMP protocol doesn't use a port)
           
  Protocol:       ICMP
           
  Origination:       Each BigFix Client will send progressive "rounds" of ICMP packets to each BigFix Relay with increasing TTLs until a BigFix Relay responds. For example, in a network of 2 BigFix Relays, one 1 hop away and one 2 hops away, the BigFix Client will send a ICMP message to both with TTL 1 and will receive 2 "time exceeded" messages from the local router. The BigFix Client will then send an ICMP message to both BigFix Relays with TTL 2 and will receive one "time exceeded" message and one reply message. BigFix Client will then choose the BigFix Relay that is one hop away.
           
  Expected Traffic Volume:       Low - The volume of the ICMP traffic is directly porportional to the number of BigFix Clients, number of BigFix Relays, and number of hops to each BigFix Relay. For example, in the example above with a BigFix Client and 2 BigFix Relays, the number of pings is 1 BigFix Client * 2 BigFix Relays * 2 hops to closest BigFix Relay = 4 ICMP pings. Each ICMP packet and the response is about 50-60 bytes. Note that if BigFix Relays are close to the BigFix Clients, the total network traffic is reduced considerably than if the BigFix Relays are far away from the BigFix Clients.
           
  Expected Traffic Frequency:       By default, the BigFix Clients attempt to find a better BigFix Relay once every 6 hours (configurable) or every time the BigFix Client is started.
           
  Possible Worst Case Traffic:      

The worst case scenario for this type of traffic is for a large number of BigFix Clients to have a large number of BigFix Relays that are very far away or for the BigFix Client to fail in some way that makes them repeatedly send ICMP packets. In such a scenario, there would be high amounts of ICMP packets being sent very quickly from the BigFix Clients to each of the BigFix Relays.

To recover from this situation, you can block ICMP at the routers or turn off BigFix Client auto-selection (can be done through BigFix in anywhere from 2 min to 2 hours depending on how busy the network is).

           
  Network Controls:       BigFix provides the following controls in the product for this type of traffic:

 
  • Relay auto-selection can be disabled
  • Configurable interval for when the BigFix Clients perform auto-selection
  • Configurable limit on the maximum number of ICMP packets to send out in a given time interval
  • Configurable limit on the maximum number of "rounds" to send out during relay auto-selection
           
  Other Notes:       The BigFix Client auto-selection can be a huge savings of time and effort because it allows the network to change (new BigFix Relays added, BigFix Relays removed, routers go up or down, etc.) and the BigFix Clients will automatically change with the network. However, if necessary, manual BigFix Relay selection can be used too.
           

 

D

           
  Network Traffic:       BigFix Console traffic
           
  Purpose:      

To administer BigFix, the BigFix Console operators will connect to the database on the BigFix Server computer.

           
  Port:      

Version 8.1 and earlier:

  • HTTP 52311 for sending actions (configurable at install time)
  • ODBC Database connection port varies (usually 1433)

Version 8.2:

  • Console initially connects to the Root Server via 52311 (by default) which then forwards the Console to HTTPS 52313 (by default) for all further interactions

Version 9.0 and later:

  • HTTPS 52311
           
  Protocol:       TCP (HTTP/HTTPS), ODBC connection-specific
           
  Origination:       The BigFix Console will connect to the database on the BigFix Server (pre-8.2) or connect to the RootServer service (8.2+)
           
  Expected Traffic Volume:       High - There is a lot of information (computer properties, vulnerabilities, etc.) sent between the BigFix Console and the database. The amount of traffic is usually measured in MB per minute, but can vary considerably depending on the size of the deployment. Note: there are usually very few BigFix Consoles active and they are usually on a high-speed connection.
           
  Expected Traffic Frequency:       Whenever a BigFix Console operator is administering BigFix, the database will repeatedly be polled for new data.
           
  Possible Worst Case Traffic:       It would be unlikely that this type of traffic would cause a "catastrophic" situation because unlike the situations involving the BigFix Clients, there are very few BigFix Consoles. If a BigFix Console user connected to the BigFix Server and repeatedly hit "F5" to refresh the data, there would be a lot of traffic between the BigFix Server and BigFix Console.
           
  Network Controls:       BigFix provides the following controls in the product for this type of traffic:



There is a "refresh rate" for each BigFix Console user (default 15 seconds)
           
  Other Notes:       If the connection is slow between the BigFix Console and the BigFix Server, it is not easy to administer BigFix so usually the BigFix Console has a high-speed connection to the BigFix Server.
           

 

E

           
  Network Traffic:       BigFix Server gets new data from external IBM Fixlet servers and download files from various download servers (such as Microsoft)
           
  Purpose:       The BigFix Server will periodically check to see if any new Fixlet messages are available from the IBM Fixlet server on the public Internet. Also, when a file needs to be downloaded, the BigFix Server will download and cache the file from a download server (for instance, all Microsoft patches are downloaded from Microsoft's download servers)
           
  Port:       80 (typical); possibly 21, 443
           
  Protocol:       TCP (HTTP/HTTPS) (some vendor downloads may require FTP)
           
  Origination:       The BigFix Server will connect to the IBM Fixlet servers or download servers on the public Internet.
           
  Expected Traffic Volume:       Low - The initial gather of all Fixlet sites will likely be less than 500 KB. Subsequent updates will usually be less than 1-2 KB of traffic. Downloads, however, can be quite large, but since the files are cached, they will be downloaded only one time.
           
  Expected Traffic Frequency:       The BigFix Server will check for new Fixlet messages once every hour by default. Files will be downloaded whenever a BigFix Console operator sends an action that requires a non-cached patch.
           
  Possible Worst Case Traffic:       It would be unlikely that this type of traffic would cause a "catastrophic" situation because unlike the situations involving the BigFix Clients, there is only one BigFix Server. If the BigFix Server were to be sent many many actions or somehow malfunctioned, it might download a lot of files from the Internet, but it seems unlikely that this would cause a problem.
           
  Network Controls:       BigFix provides the following controls in the product for this type of traffic:



There is a configurable interval that the BigFix Server will check for new Fixlet messages.
           
  Other Notes:       The BigFix Server is the only component in a default BigFix installation that contacts the Internet directly.
           





More information about BigFix Relays can be found here and more general documentation about BigFix can be found here.

Note that port 52310 is used by the BigFix Server to send updates between components.

For more information on how Mobile Device Management (MDM) is designed to work in your environment, see the MDM architecture.