Bandwidth Throttling

2 likes Updated 4/3/14 9:10 AM by GreenEagleLeaderTags:

Overview of Bandwidth Throttling

Note:  The information presented in this article, is currently, a little bit out of order.  We are working to re-arrange the content in this article to be more ordered and concise.

 

Many network environments have limited bandwidth between certain geographic locations, between offices and home users using VPNs, etc. Deploying large Microsoft patches (for example: the Windows XP SP2 update is 275 MB!) can easily overwhelm limited bandwidth connections and cause bandwidth problems for certain users or applications.

 

To avoid these types of problems, IEM provides a number of mechanisms to reduce bandwidth usage including; a properly implemented and maintained IEM Relay architecture, distributing patch deployments over time, and the use of bandwidth throttling configurations between IEM components. Through client settings, the IEM Console operator has the ability to set the maximum number of bytes per second that will be used to send files over a network connection. IEM can be configured to throttle bandwidth at the IEM Server, IEM Relay, or IEM Client component levels. Below are instructions on how to configure these IEM components to use bandwidth throttling.

 

Important Note: Starting in BigFix/TEM/IEM version 5.0 the bandwidth throttling configuration settings were moved to tasks within the BES Support site.  This provides a more intuitive and explicit way of setting these configurations (versus the more error prone way of having to use custom settings).

 

To see these tasks in the console:  On left hand navigation tree drill down under, All Content > Fixets and Tasks > Tasks Only > By Site > BES Support and search for the term throttl in the search box:

 

Note: all settings are set as registry key client settings on the endpoint in the client section of the registry (HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BigFix\EnterpriseClient\Settings\Client) and with a String [REG_SZ] type of value named value. The value is then going to be entered as a numeric (for example: 500 bps is entered as 500).

Static Throttling:

151 - BES Server Setting: Throttle Outgoing Download Traffic

152 - BES Relay Setting: Download Throttling

163 - BES Relay Setting: Throttle Outgoing Download Traffic

167 - BES Client Setting: Download Throttling

Dynamic Throttling:

457 - BES Client Setting: Dynamic Download

458 - BES Relay Setting: Dynamic Download Throttling

459 - BES Relay Setting: Dynamically Throttle Outgoing Traffic

462 - BES Server Setting: Dynamically Throttle Outgoing Traffic

605 - BES Client Setting: Enable/Disable Dynamic Throttling

702 - BES Relay/Server Setting: Enable/Disable Dynamic Throttling

 

There is also an Analysis in the BES Support site (# 218 Bandwidth Throttling Status), All Content > Analyses > By Site > BES Support that can be activated to display the bandwidth throttling status settings for each type of component/segment in the deployment (Client, Relay, and Server), the analysis contains the following retrieved properties:

Static Throttling:

BES Client Download Throttling

BES Relay Download Throttling

BES Relay Total Outbound Throttling

BES Server Total Outbound Throttling

Dynamic Throttling:

BES Client Dynamic Download Throttling: (min - max) percentage

BES Relay Dynamic Download Throttling: (min - max) percentage

BES Relay Total Outbound Dynamic Throttling: (min - max) percentage

BES Server Total Outbound Dynamic Throttling: (min - max) percentage

 

Refer to the following diagram to determine which settings configure bandwidth throttling for that specific segment type in the Methods of Bandwidth Throttling section below:

 

Methods of Bandwidth Throttling



Throttling through IEM Relays (A)



IEM Relays downloading from the IEM Server

IEM Relays can be configured to throttle file downloads when downloading from the IEM Server. When IEM Relay throttling is enabled, the IEM Relay will download from the IEM Server no more than the specified number of bytes per second. This setting is especially useful for IEM Relays that have a slow connection to the IEM Server (such as a IEM Relay in a remote location connected by a 56 kbps modem).

Configuration: IEM Client throttling can be configured by using the "_BESGather_Download_LimitBytesPerSecond" setting on the IEM Relays. Information on how to configure this setting (and client settings in general) can be found here.

Note: When the Client asks the Relay "please tell me the latest contents of site X" (logged as GatherActionMV command received. Version difference, gathering), the interaction is not throttled. The response of the Relay is typically small (anywhere from 0-~40k). If absolutely necessary, you can turn down the gather intervals on Clients to get this information less frequently, but this traffic should usually be negligible.

 

Total outgoing download traffic for the IEM Relay

The IEM Relay in BigFix 4.0 and above can be configured to throttle the cumulative file downloads at any given time. When this throttling setting is enabled, an IEM Relay will send out no more than the specified number of bytes per second for all file downloads (including IEM Clients and child IEM Relays). This setting is especially useful if there is a concern in a local area network that too much bandwidth will be used when a patch is sent simultaneously to many IEM Clients.

Configuration: IEM Relay cumulative download throttling can be configured by using the "_BESRelay_HTTPServer_ThrottleKBPS" setting on the IEM Relay. Set this number to the total number of kilobytes that the IEM Relay will give to all of the IEM Clients combined per second.



Throttling through IEM Clients (B)



IEM Clients downloading files from the IEM Server or IEM Relay

BigFix Clients can be configured to throttle file downloads when downloading from the BigFix Server or BigFix Relays. When BigFix Client throttling is enabled, a BigFix Client will download from the BigFix Server or BigFix Relays at no more than than the specified number of bytes per second. This setting is especially useful for individual computers that are on slow connections (such as travelling sales representatives or home users on dial-ups).

Configuration: BigFix Client throttling can be configured by using the "_BESClient_Download_LimitBytesPerSecond" setting on the BigFix Clients. Information on how to configure this setting (and client settings in general) can be found here.



Throttling through the IEM Server/Relays (C/D)



Total outgoing download traffic for the IEM Server/Relays

The IEM Server can be configured to throttle the cumulative file downloads at any given time. When this throttling setting is enabled, the IEM Server/Relays will send out no more than the specified number of bytes per second for all file downloads (including IEM Clients and child IEM Relays). This setting is especially useful if there is a concern in a local area network that too much bandwidth will be used when a patch is sent out to many IEM Clients simultaneously.

Configuration: IEM Server cumulative download throttling can be configured by using the "_BESRelay_HTTPServer_ThrottleKBPS" setting on the IEM Server. Information on how to configure this setting can be found here.

This information is current as of BigFix 7.2. Before using these settings, you may want to check for any changes in IEM Console/Server Settings or IEM Client Settings. Please update this document if appropriate. Note that BES Root Server or BES Relay services require a restart before any changes to these settings will take effect.

 

What Can Be Throttled?

IEM has a great number of different settings that control how different components throttle traffic between each other. The following goal is give a comprehensive view of all the throttling options.
 
A common misconception is that various "bandwidth throttling" features in IEM are designed to throttle all IEM traffic between any two machines. In fact, throttling is always done for specific components, and many components have no throttling available at all.
 

Throttled Components

These are any files that are downloaded directly from the wwwrootbes directory of the Relay/Server/WebReports. Files downloaded for actions (e.g. patches, service packs, etc.) are downloaded this way. The contents of Fixlet sites are also downloaded this way during the gather interaction (although the site directory listing comes in separately). Web Reports also serves up "support" files (such as SWFs) this way, although really, why are you trying to throttle Web Reports?
 
 

Unthrottled Components

 
There are however, a number of "throttle-like" capabilities around client registration. The amount of downstream UDP traffic generated can be rate limited on the Relay/Server side, the frequency of client registration can be turned down on the client side, and the amount of ICMP traffic generated during relay selection can be rate limited on the client side. Note that client registration is not bandwidth intensive, although it can be a significant source of load.
 
Download request plugin, status reporting plugins, the "Web Reports" plug-in, etc. For the Client, we expect this traffic to be negligible. However, things like the download status report generate a lot of traffic between the Console and the Root Server, and they cannot be throttled.
 
When the Client asks the Relay "please tell me the latest contents of site X" (logged as 'GatherActionMV command received. Version difference, gathering'), the interaction is not throttled. The response of the Relay is typically small (anywhere from 0-~40k). If absolutely necessary, you can turn down the gather intervals on Clients to get this information less frequently, but this traffic should usually be negligible.
 
There are also things you can do to control the amount of traffic going upstream through the "PostResults" interface. The most basic "throttling" mechanism is simply to turn up the minimum report interval on the Client or lengthen the heartbeat interval. You can also put a crude limit on the amount of traffic a Relay can send up through a combination of the "ResultSizeLimit" and "ResultTimeLimit" Relay settings. However, you should check with Customer Support before doing so: it's unlikely that you'll get the behavior you expect. Posting results takes more bandwidth than client registration, but is still much smaller than the amount of bandwidth used by the download and upload components. For most deployments, the amount of traffic should be negligible.
 

Throttling Minimum Transfer Rates

During throttled communication, the IEM Client will send chunks of data and then wait longer than necessary before sending the next chunk. The IEM Client can vary the amount of data in each chunk along with the amount of time to wait. By lowering the amount of data per chunk and maximizing the amount of data between chunks, a minimum transfer rate will be established for throttling.
 
Dynamic throttling and static throttling systems share the code that creates the minimum transfer rate so these thresholds are the same. The introduction of dynamic throttling ended up raising the minimum transfer rate and then we had to lower it again due to customer feedback, so the minimum throttling rate has changed for different versions of the product.
 
The IEM Relay component actually does the throttling work so if you need to upgrade IEM to change the minimum throttling rate, it is important to upgrade the IEM Relays along with the IEM Clients to the newer version.
 
The effective minimum throttling rates by version are:
  • 6.0 and below: 20-30 Bytes/Sec
  • 7.0, 7.1 less than 7.1.8.1, and 7.2 less than 7.2.1.269: 4096 Bytes/Sec
  • 7.1 at or above 7.1.8.1 and 7.2 at or above 7.2.1.269: 100 Bytes/Sec
 
All throttling settings are given in either BPS (Bytes per Second) or KBPS (KBPS). Given the effective minimum values listed above, if you set the throttling rate below the effective minimum it doesn't go that low, it only goes to the effective minimum value. Also notice that the effective minimum is less than 1 KBPS (in most versions) but the minimum value you can set KBPS throttling setting to is only 1 so you can't reach the effective minimum. IE., in some cases the effective minimum rate is controlled by the setting and in other cases the effective minimum is controlled by the client's throttling limits.
 

Upload Throttling

The uploads generated by the "Upload Manager" can be throttled from the Client side or from the Relay/Server side. This component only supports static throttling. There is a "PostFile" setting that sets an overall throttle on incoming connections, and an "UploadManager" setting that sets the throttle on outgoing connections (there is only one outgoing connection at a time). When both are set, the child is responsible for using the lesser of the two values.
  • _BESRelay_PostFile_ThrottleKBPS
    • "0" means "no limit" (Default: 0)
    • This setting is not sufficient to throttle PostFile, indeed it needs _BESClient_UploadManager_ThrottleKBPS to be set on Client with a non-0 value to work. The limit will be the lesser of the two values.
    • At the start of an upload interaction, the PostFilePlugIn divides this number by the total number of uploads currently in progress on the Relay and sends the result down to the child, who is responsible for respecting the resulting limit.
    • Same setting on both Server and Relay
  • _BESRelay_UploadManager_ThrottleKBPS
    • Default: 0
    • Only relevant on Relay (Server has no one to upload *to*, although DSA may change that at some point)
    • When Relay uploads files to its parent, it will limit itself to this rate. It provides the limitation by breaking the interaction into chunks and doing a connection for each chunk, with waits in between (this in contrast to download throttling, which maintains a single connection over the length of the interaction).
    • "0" means "no limit"
  • _BESClient_UploadManager_ThrottleKBPS
    • Default: 0
    • Only relevant on Client
    • When Client uploads files to its parent, it will limit itself to this rate. It provides the limitation by breaking the interaction into chunks and doing a connection for each chunk, with waits in between (this in contrast to download throttling, which maintains a single connection over the length of the interaction).
    • "0" means "no limit"

These are files uploaded from the endpoints through the "Upload Manager".

Download throttling settings have two major axes:

  • "Server Side" or "Client side"
    • Server-side throttling (Server/Relay/Web Reports) is expressed as an amount of bandwidth to be shared among all connecting children
    • Client-side throttling (Client/Relay) is expressed as an amount of bandwidth to be used on a single upstream connection. Note that Clients may use more bandwidth if they have multiple simultaneous upstream connections.
  • Static or Dynamic
When Server-side and Client-side throttling is in effect, IEM components use the lower of the calculated bandwidth limits. When Dynamic and Static throttling are both in effect, the Dynamic throttling settings are used in place of the Static throttling settings.
 
Server-side static throttling is the only type of throttling that can affect non-IEM components (such as Web Browsers).
 
 

Static Throttling

Server Side

Server-side static throttling settings control the total amount of download traffic that a server will send out to Clients using static throttling. The amount of bandwidth allocated to any given write connection is simply the "ThrottleKBPS" setting divided by the number of active write connections. Note that plug-in connections do not count as "write" connections. However, file downloads with static or dynamic throttling enabled do count as "write" connections.
 
If you have:
  • ThrottleKBPS = 500
  • One Client connecting without dynamic throttling
  • One Client connecting with dynamic throttling

...then the Client without dynamic throttling will get 250 KBPS of bandwidth allocated. The bandwidth usage of the Client with dynamic throttling will be determined by the dynamic throttling algorithm -- it may turn out to be much less or greater than 250 KBPS, so that the total bandwidth usage of the server will not necessarily be 500 KBPS.

Note:  Server-side settings are in KBPS.
 
For Relay and Root Server:
  • _BESRelay_HTTPServer_ThrottleKBPS
    • Default: 0
    • "0" means "no limit"
  • For Web Reports:
    • _WebReports_HTTPServer_ThrottleKBPS
    • Default: 0
    • "0" means "no limit"

 

Client Side

Client-side static throttling is the simplest of these settings. A "Client" (could be a Client or a Relay) simply tells its parent "please send me files at this speed" and the parent obliges. Settings are in BPS.
 
For a BES Client:
  • _BESClient_Download_LimitBytesPerSecond
    • Default: 0
    • "0" means "no limit"

 

For a BES Relay downloading files from its parent:
  • _BESGather_Download_LimitBytesPerSecond
    • Default: 0
    • "0" means "no limit"

 

Throttle Groups

"Throttle Groups" are a part of the static throttling functionality that allow a set of Clients to throttle themselves as a group instead of individually (or along with every other connection through Server-side throttling).
 
When a Client identifies itself as part of a "throttle group", it sends up the name of the group it belongs to, along with the speed it would like the entire group to have. So for example a Client might say "I'm in the 'remote' group, and we'd like to be given 10000 BPS as a whole". When the Server sends data down to that Client, it throttles based on the total number of connections in the Clients group. So if there are five active connection from the 'remote' group, our Client will get 2000 BPS. Note that different Clients can send up different values for the "limit bytes per second", so another Client could say "I'm in the 'remote' group, and we'd like to be given 5000 BPS as a whole", and it would be given 1000 BPS at the same time our first Client was given 2000 BPS.
 
The special group "ipaddress" will cause the Server to group this connection along with other connections from the same IP address. This is the default for Relay upstream traffic. Clients default to the group "", so that their "LimitBytesPerSecond" setting is shared among all of their currently active file downloads.
 
  • _BESGather_Download_ThrottleGroup (valid on Windows Server only)
    • Default: "ipaddress"
    • The parent will consider this Relay to be part of whichever group is specified here
  • _BESClient_Download_ThrottleGroup
    • Default: computer id as string
    • The parent will consider this Client to be part of whichever group is specified here
    • This is a string value: older versions of ClientSettings documentation incorrectly claimed this was a numeric value.
Dynamic throttling is unaffected by throttle groups (an interesting side effect of this is that a Client set to target 20% of available bandwidth may end up using 40% if it's downloading two files simultaneously).
 

Dynamic Throttling

A NOTE ON DYNAMIC DOWNLOAD THROTTLING:
 
A dynamic bandwidth throttling implementation is not something currently recommended as there still exists the same unreliable bandwidth calculations as when first developed. These bandwidth calculations are currently not reliable or predictable from one network environment to another. This is especially true in low bandwidth network environments.
 
It is recommended that you configure static bandwidth throttling (see section above) and then use other more reliable and configurable networking appliances, network traffic shapers, etc. to manage the bandwidth allocation dynamically within your deployment's network.
 
If you would still like to use dynamic bandwidth throttling in your deployment; please contact our Professional Services group for an engagement in implementing and testing dynamic bandwidth throttling within your deployment and network.

 

Dynamic throttling is co-operative: this means it must be turned on for both the server-side and client-side in order for any throttling to happen. Once you have turned on dynamic bandwidth throttling, IEM components will try to calculate the amount of "available" bandwidth on the link they are using. They will then use a percentage of the bandwidth that you specify, with minimum and maximum bounds.

 

Server Side

Server-side dynamic throttling settings control the total amount of download traffic that a server will send out to Clients using dynamic throttling. The "min", "max", and "percentage" settings are all divided by the number of active write connections. Note that plug-in connections do not count as "write" connections. However, file downloads with static or dynamic throttling enabled do count as "write" connections.
 
If you have:
  • DynamicThrottlePercentage = 50
  • One Client connecting without dynamic throttling
  • One Client connecting with dynamic throttling

 

...then the Client with dynamic throttling will get 25 percent of estimated bandwidth allocated. Note that this is an estimate of the bandwidth between that specific Client and the server, so another Client with the same settings might end up with a different amount of bandwidth. There is no way to say "use 50% of all outgoing bandwidth" because the server may have different bandwidth availability for different connections. However, if 10 Clients are all connecting over the same thin pipe, and the server is set to target "20%" usage as a whole, than it will target "2%" for each of the ten Clients and should end up at roughly "20%" usage for the pipe.
 
Server-side settings are in KBPS:
  • _BESRelay_HTTPServer_DynamicThrottleEnabled
    • Default: 0
    • non-zero = "enabled"
  • _BESRelay_HTTPServer_DynamicThrottleMaxKBPS
    • Default: 0
    • "0" means "no limit"
  • _BESRelay_HTTPServer_DynamicThrottleMinKBPS
    • Default: 0
  • _BESRelay_HTTPServer_DynamicThrottlePercentage
    • Default: 0
    • specify a target percentage from 1 to 100 (0 is treated the same as 100).

 

Client Side

Settings are in BPS.
 
For an IEM Client:
  • _BESClient_Download_DynamicThrottleEnabled
    • Default: 0
    • non-zero = "enabled"
  • _BESClient_Download_DynamicThrottleMaxBytesPerSecond
    • Default: 0
    • "0" means "no limit"
  • _BESClient_Download_DynamicThrottleMinBytesPerSecond
    • Default: 0
  • _BESClient_Download_DynamicThrottlePercentage
    • Default: 0
    • specify a target percentage from 1 to 100 (0 is treated the same as 100).

 

For a BES Relay downloading files from its parent:
  • _BESGather_Download_DynamicThrottleEnabled
    • Default: 0
    • non-zero = "enabled"
  • _BESGather_Download_DynamicThrottleMaxBytesPerSecond
    • Default: 0
    • "0" means "no limit"
  • _BESGather_Download_DynamicThrottleMinBytesPerSecond
    • Default: 0
  • _BESGather_Download_DynamicThrottlePercentage
    • Default: 0
    • specify a target percentage from 1 to 100 (0 is treated the same as 100).
 

Throttling Internet Traffic from TEM through a Cisco Router

As an An alternative to using throttling provided by IEM,  consider configuring your network routers and segments to manage the allowable bandwidth usage on your network .IEM does not provide the ability to throttle the upstream connection from IEM  server to external sites (eg. for downloading patches), however most router/firewall/switches that do rate-limiting should be able to accomplish this instead.

eg. On a Cisco router the following policy can be used
 
---- cut here ----

access-list 10 permit host w.x.y.z
 
class-map match-any SpecificHost
match access-group 10
.
policy-map SqueezeHost
class SpecificHost
police 128
.
interface e0/0
service-policy input SqueezeHost
service-policy output SqueezeHost
---- cut here ----
.
This limits the Internet side bandwith to 128K max