Bandwidth Allocation in Virtual Server Instances

When a VSI is provisioned in the IBM Cloud Virtual Private Cloud (VPC), a fixed amount of bandwidth is allocated to the VSI. The allocated amount of bandwidth:

• Is determined by the instance profile selected when provisioning the VSI.
• Is not user changeable. The amount of bandwidth allocated to a VSI can only be changed by doing the following:
  • Choosing a different instance profile that offers a more suitable amount of bandwidth when provisioning the VSI.
  • Resizing the VSI to use an instance profile that offers more bandwidth.

All network and block storage volume bandwidth requirements for the VSI are satisfied from the allocated bandwidth. Bandwidth is stated in units of megabits per second (Mbps) or gigabits per second (Gbps).

In general, the allocated amount of bandwidth is currently 2 Gbps per virtual CPU (vCPU) up to a maximum of 80 Gbps for most instance profiles (although for the Ultra High Memory instance profiles, the allocated bandwidth is 1 Gbps per vCPU). The allocated bandwidth for each profile is given under the “Bandwidth Cap” column in the documentation for Instance Profiles.

The amount of bandwidth allocated to the VSI is divided for use by networking (“network bandwidth”) and block storage volumes (“volume bandwidth”). Network bandwidth includes the traffic that flows over the VSI’s network interfaces, while volume bandwidth includes traffic to the VSI’s block storage volumes. Traffic to NFS-mounted VPC File Shares is included in the network bandwidth because the NFS traffic flows over the VSI’s network interfaces. Access to local disks (Instance Storage) does not consume any of the bandwidth allocated to the VSI.

By default, the network bandwidth allocation is 75% of the instance’s allocated bandwidth, while the volume bandwidth allocation is 25%. Modifying the VSI’s allocated bandwidth means changing the network/volume bandwidth split to a different percentage. Increasing volume bandwidth decreases network bandwidth while decreasing volume bandwidth increases network bandwidth.

For example, a bx2-2×8 balanced server profile is entitled to 4,000 Mbps of bandwidth. By default, 3,000 Mbps is allocated to networking and 1,000 Mbps is allocated to storage. When provisioning a VSI via the IBM Virtual Private Cloud UI (UI), the actual volume and network bandwidth values are displayed for the selected profile. Selecting the bx2-2×8 profile while creating a VSI via the UI shows the following:

The amount of bandwidth allocated to the VSI, the current network bandwidth and the current volume bandwidth can be displayed using the IBM Virtual Private Cloud CLI (CLI). Only the relevant JSON fields are shown below:

$ ibmcloud is instance <instance_id> --json
{
...
"bandwidth": 4000,
"total_network_bandwidth": 3000,
"total_volume_bandwidth": 1000,
...
}

The values can also be displayed via the UI by navigating to the Instance details view for the desired instance:

Network bandwidth is the amount of bandwidth assigned exclusively for use by the VSI’s network interfaces. The network bandwidth limit applies to each direction (inbound and outbound) separately.

Network bandwidth is set, implicitly, when the VSI is created or as a result of a change in volume bandwidth — it cannot be set explicitly. Network bandwidth is shared equally by all attached network interfaces, but any one network interface is capped at a maximum of 16,000 Mbps (16 Gbps) at the time of publication, provided the selected instance profile allocates enough total bandwidth to allow it. Leftover network bandwidth beyond that currently allocated to attached network interfaces cannot be used unless one or more additional network interfaces are attached to the VSI.

For example, a bx2-16×64 balanced server profile is entitled to 32,000 Mbps of bandwidth, of which 24,000 Mbps (75%) is allocated to networking by default. If only one network interface is attached to the VSI, it will be allocated 16,000 Mbps of bandwidth (the maximum allowed for one interface), leaving 8,000 Mbps of unused network bandwidth. If a second network interface is attached to the VSI, each interface will be allocated 12,000 Mbps of bandwidth because the available 24,000 Mbps will be allocated equally across both interfaces. Attaching a third network interface will result in each of the three interfaces being allocated 8,000 Mbps.

Using the CLI, the following command can be used to display the bandwidth allocated to the primary network interface — when there is only one attached — for a VSI using the bx2-16×64 profile and provisioned using the default amount of network bandwidth of 24,000 Mbps:

$ ibmcloud is instance-network-interface <instance_id>
 <primary_interface_id>...
ID 0727-4777dcc2-af8f-414d-8325-16898708dd26
Name     primary
Type     primary
Speed    16000
...

After attaching a second network interface to the VSI, the network bandwidth will now be divided equally between the primary and the newly attached interface. Checking both interfaces, the allocated bandwidth is now 12,000 Mbps for each:

$ ibmcloud is instance-network-interface <instance_id> <primary_interface_id>

...
ID 0727-4777dcc2-af8f-414d-8325-16898708dd26
Name     primary
Type     primary
Speed    12000
...


$ ibmcloud is instance-network-interface <instance_id> <interface2_id>

...
ID 0727-54245510-be5f-48f2-93e3-97df1eb9c1af
Name     interface2
Type     secondary
Speed    12000
...

Volume bandwidth is the amount of bandwidth assigned exclusively for use by the VSI’s block storage volumes. Volume bandwidth limits apply to the sum of read and write volume traffic.

Block storage volumes have an “unattached” bandwidth value associated with them. The value represents the maximum bandwidth assigned to the volume. The value is calculated at volume provisioning time (or when volume IOPS or capacity are modified) and is a function of the current IOPS for the volume, the volume capacity and the volume profile. Unattached bandwidth is calculated at a block size of 16 KB for the custom, general-purpose and 5iops-tier profiles and at a block size of 256 KB for the 10iops-tier profile.

Block storage volumes also have an “attached” bandwidth value associated with them. The value represents the actual amount of bandwidth available to the volume when it is attached to a VSI. It is a function of the volume’s unattached bandwidth, the unattached bandwidth of all currently attached volumes and the amount of volume bandwidth set for the VSI. The attached bandwidth value is calculated for all volumes currently attached to a VSI when any of following happens:

• A new volume is attached to the VSI.
• A volume is detached from the VSI.
• A volume is resized (capacity or IOPS).
• The VSI’s volume bandwidth amount is modified.

The sum of read bandwidth and write bandwidth for a volume cannot exceed the volume’s attached bandwidth.

The VSI’s volume bandwidth is divided among all volumes attached to the VSI. The bandwidth for each attachment is calculated as follows:

• The boot volume’s attachment bandwidth is fixed at 393 Mbps to ensure reasonable boot times. 393 Mbps is the result of multiplying 3,000 (the IOPS assigned to boot volumes) by 16 KB (the block size used when calculating the bandwidth for general-purpose volumes).
• The remaining amount of volume bandwidth (the VSI’s volume bandwidth amount minus the 393 Mbps assigned to the boot volume) is then assigned to each of the other volumes attached to the VSI as follows:
  • If the sum of the remaining volumes’ unattached bandwidth is less than or equal to the remaining amount of volume bandwidth, then the attachment bandwidth for each remaining volume is set to its unattached bandwidth.
  • If the sum of the remaining volumes’ unattached bandwidth is greater than the remaining amount of volume bandwidth, then, for each remaining volume, the attachment bandwidth is calculated to be proportional to the volume’s unattached bandwidth, so that the sum of all volume attachments bandwidth, including the boot volume, equals the total volume bandwidth allocated to the VSI.
• There can be unused volume bandwidth if the total volume bandwidth allocated for the VSI is larger than the sum of the unattached bandwidths of all volumes attached to the VSI.

Because attachment bandwidth is re-calculated every time a volume is attached, attaching additional volumes can reduce the bandwidth allocated to volumes that were already attached, while detaching volumes can increase the attachment bandwidth allocated to the volumes that remain attached.

Also, because attachment bandwidth is assigned per volume attachment and it is not shared by other attachments, there can be times where some volumes are not using all the bandwidth allocated to them, while heavily used volumes are exhausting the bandwidth allocated to them but cannot make use of the lightly used volume’s spare bandwidth.

It is important to carefully consider the volume bandwidth requirements and adjust the VSI’s volume bandwidth value accordingly to provide an acceptable level of service to all volumes attached to the VSI.

The bandwidth allocation for the VSI can be modified to apportion network and volume bandwidth values differently, in order to meet workload requirements.

The bandwidth allocation can be modified at VSI provision time, but only if the VSI is provisioned using the IBM Virtual Private Cloud CLI (CLI) or the IBM Virtual Private Cloud API (API) — it is not possible to modify the bandwidth allocation if the VSI is provisioned via the IBM Virtual Private Cloud UI (UI).

The following CLI command can be used to provision an VSI using the bx2-2×8 profile and override the bandwidth allocation to allocate 2000 Mbps each for network and volume bandwidth:

$ ibmcloud is instance-create bw-allocation-vsi1 <vpc_id> <zone>
bx2-2x8 <subnet_id> --image <image_id> --keys <key_id>

Notice that only thetotal-volume-bandwidth value can be set. Doing so implicitly sets the network bandwidth.

The bandwidth allocation can also be modified dynamically any time after the VSI has been provisioned via the UI, the CLI and the APIs. Bandwidth can be adjusted to any value, but both network and volume bandwidth must be at least 500 Mbps.

The following CLI command can be used to modify the current bandwidth allocation for the VSI created in the previous example to 2,500 Mbps network and 1,500 Mbps volume bandwidth:

$ ibmcloud is instance-update <instance_id> --total-volume-bandwidth 1500
...
Bandwidth(Mbps)         4000
Volume bandwidth(Mbps)  1500
Network bandwidth(Mbps) 2500
...

Using the UI, navigate to the Instance details view for the desired VSI and click on the edit (pencil) button next to Bandwidth allocation:

That brings up the Edit bandwidth allocation dialog where the Volume bandwidth value can be modified as desired.

When resizing a VSI to a smaller instance profile (one with fewer vCPUs and thus less instance bandwidth) it may be necessary to adjust the VSI’s volume bandwidth amount prior to the resize operation. In order to successfully resize, the VSI’s volume bandwidth must be at least 500 Mbps less than the allocated bandwidth amount in the target instance profile.

For example:

• Current profile: mx2-8×64. Total bandwidth: 16,000 Mbps. 12,000 Mbps allotted to network and 4,000 Mbps to volume, which is the default allotment for the instance profile.
• Target profile: bx2-2×8. Total bandwidth: 4,000 Mbps. 3,000 Mbps allotted to network and 1,000 to volume, which is the default allotment for the instance profile.

The resize operation will fail because the current amount of volume bandwidth (4,000 Mbps) is not at least 500 Mbps less than the target instance profile’s total bandwidth (4,000 Mbps), which would leave zero (0) bandwidth for network. In order to successfully resize, the VSI’s volume bandwidth amount must be adjusted to 3,500 Mbps or less prior to attempting the resize operation.

Attached data volumes remain attached in the resized VSI but their attachment bandwidth amount(s) will be re-calculated, possibly changing the values from the pre-resize amounts.

Continuing with the bx2-2×8 VSI shown above, with the volume bandwidth allocation set at 1,500 Mbps, two data volumes will be created and attached to show what happens to each volume attachment after various attachment scenarios and after a change in volume bandwidth for the VSI. Responses to the CLI commands will only show relevant fields, not the entire responses.

The data/secondary volumes to be used are as follows:

• 1,500 GB, 5iops-tier volume (7,500 IOPS, 983 Mbps unattached bandwidth)
• 4,000 GB, 5iops-tier volume (20,000 IOPS, 2,621 Mbps unattached bandwidth)

Because the VSI’s volume bandwidth is currently set to 1,500 Mbps, 393 Mbps of which is assigned to the boot volume, there will only be 1,107 Mbps of volume bandwidth available to allocate to the two volumes once they’re attached.

First, attach the 1,500 GB volume:

$ ibmcloud is instance-volume-attachment-add
bw-allocation-attachment1 <instance_id> <volume1_id>

Then display the volume attachment:

$ ibmcloud is instance-volume-attachment <instance_id> <attachment1_id>

Getting volume attachment 0727-7a94d3d8-45fd-4e67-864d-a7d5720cb024 ...
ID                  0727-7a94d3d8-45fd-4e67-864d-a7d5720cb024
Name                bw-allocation-attachment1
Status              attached
Bandwidth(Mbps)     983
Type data
...

As seen above, the attachment has been assigned 983 Mbps of bandwidth (the same amount as the volume’s unattached bandwidth) because the VSI’s remaining volume bandwidth was 1,107 Mbps. After the volume is attached, the remaining amount of volume bandwidth is 124 Mbps (1,107 – 983).

Now attach the 4,000 GB volume:

$ ibmcloud is instance-volume-attachment-add bw-allocation-attachment2 <instance_id> <volume2_id>

Then display the volume attachment:

$ ibmcloud is instance-volume-attachment <instance_id> <attachment2_id>                                      Getting volume attachment 0727-a16f3d6c-9dec-433e-a052-4876765e3ba1 ...                                      ID                0727-a16f3d6c-9dec-433e-a052-4876765e3ba1                                                  Name              bw-allocation-attachment2                                                                  Status            attached                                                                                   Bandwidth(Mbps)   805                                                                                        Type              data...

Remembering that attachment bandwidth is recalculated for all attached volumes whenever a new volume is attached. We see that although only 124 Mbps of volume bandwidth remained available, attachment2 was assigned 805 Mbps of bandwidth. That means the bandwidth for attachment1 had to be reduced from the 983 Mbps initially assigned to it:

$ ibmcloud is instance-volume-attachment <instance_id> <attachment1_id>                                      Getting volume attachment 0727-7a94d3d8-45fd-4e67-864d-a7d5720cb024 ...                                      ID                0727-7a94d3d8-45fd-4e67-864d-a7d5720cb024                                                                  Nam               bw-allocation-attachment1                                                                                                            Status            attached                                                                                              Bandwidth(Mbps)   301                                                                                          Type              data ...

The two data volumes were allocated attachment bandwidth in proportion to their unattached bandwidths: 805 Mbps / 2,621 Mbps = 301 Mbps / 983 Mbps

The boot volume attachment’s allocated bandwidth is unchanged:

$ ibmcloud is instance-volume-attachment <instance_id> <bootvol_attachment_id> Getting volume attachment 0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716 ...                                                                    ID                 0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716                                                                 Name               rental-linguini-showman-staunch                                                                         Status             attached                                                                                              Bandwidth(Mbps)    393                                                                                          Type boot ...

If attachment1 is now deleted, it should free up 301 Mbps of bandwidth. Because attachment bandwidth is also recalculated for all attached volumes whenever a volume is detached, we should expect the amount of bandwidth currently assigned to attachment2 to increase:

$ ibmcloud is instance-volume-attachment-detach <instance_id> <attachment1_id>

Display attachment2 and the boot volume attachment:

$ ibmcloud is instance-volume-attachment <instance_id> <attachment2_id>                                        Getting volume attachment 0727-a16f3d6c-9dec-433e-a052-4876765e3ba1 ...                                                                           ID                  0727-a16f3d6c-9dec-433e-a052-4876765e3ba1                                                                 Name                bw-allocation-attachment2                                                                               Status              attached                                                                                              Bandwidth(Mbps)     1107                                                                                         Type data ...                                                                                                $ ibmcloud is instance-volume-attachment <instance_id> <bootvol_attachment_id>                                       Getting volume attachment 0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716 ...                                      ID                  0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716                                                                 Name                rental-linguini-showman-staunch                                                                         Status              attached                                                                                              Bandwidth(Mbps)     393                                                                                          Type boot ...

As seen above, the amount of bandwidth allocated to attachment2 increased to use all available volume bandwidth, and the boot volume’s attachment bandwidth remains unchanged.

Finally, what happens when the VSI’s volume bandwidth is reduced from 1,500 Mbps to 750 Mbps?

$ ibmcloud is instance-update <instance_id> --total-volume-bandwidth 750                                     Updating instance 0727_4a6c774b-b39b-4221-a5bd-787caeca7935 ...                                              ID                          0727_4a6c774b-b39b-4221-a5bd-787caeca7935                                                                 Name                        bw-allocation-vsi1 ...                                                                                  Status                      running                                                                                               Profile                     bx2-2x8                                                                                                       vCPUs                       2                                                                                                      Memory(GiB)                 8                                                                                                                     Bandwidth(Mbps)             4000                                                                                         Volume bandwidth(Mbps)      750                                                                                            Network bandwidth(Mbps)     3250 ...

Display attachment2 and the boot volume attachment:

$ ibmcloud is instance-volume-attachment <instance_id> <attachment2_id>                                                Getting volume attachment 0727-a16f3d6c-9dec-433e-a052-4876765e3ba1 ...                                      ID              0727-a16f3d6c-9dec-433e-a052-4876765e3ba1                                                                   Name            bw-allocation-attachment2 ...                                                                           Status          attached                                                                                              Bandwidth(Mbps) 357                                                                                          Type            data ...                                                                                                $ ibmcloud is instance-volume-attachment <instance_id> <bootvol_attachment_id>                               Getting volume attachment 0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716 ...                                      ID                 0727-f7f0eaa5-eca5-4267-b34a-1d9d509d5716                                                                 Nam                rental-linguini-showman-staunch ...                                                                     Status             attached                                                                                               Bandwidth(Mbps)    393                                                                                          Type               boot ...

Because attachment bandwidth is also recalculated for all attached volumes whenever the VSI’s bandwidth allocation is modified, and because the new volume bandwidth value is lower than the previous one, the amount of bandwidth currently allocated to attachment2 is reduced from 1,107 Mbps to 357 Mbps, while the boot volume’s attachment bandwidth allocation remains unchanged.

In this post, we described how a fixed amount of bandwidth is allocated to a VSI based on the instance profile used to provision the VSI, how the allocated amount of bandwidth is divided for use by network and block storage volumes, what the default apportionments are and how the apportionments can be modified.

We also described how network and volume bandwidth are assigned to attached network interfaces and volumes. The key takeaways are as follows:

• There is a fixed amount of bandwidth allocated to a VSI.
• The allocated amount is divided for use by network and block storage volumes. The default network/volume bandwidth apportionment, of 75% network and 25% volume, can be modified at VSI provision time or at any time after that. Changes take effect immediately.
• Neither network nor block storage volumes can use the other’s bandwidth allotment.
• Network bandwidth is divided equally between all attached interfaces and is recalculated every time a network interface is attached to or detached from the VSI.
• Volume bandwidth is divided proportionally, based on the volume’s unattached bandwidth. It is recalculated every time a volume is attached/detached/resized and when the VSI’s network/volume bandwidth allotment is modified. It is assigned per attached volume, not shared among all attached volumes.

For more information about bandwidth allocation see, “Bandwidth allocation for instance profiles” and “Bandwidth allocation for block storage volumes.”

Learn more about IBM Cloud VPC.