The purpose of traffic shaping

With the ever-increasing demands on network systems for a number of protocols, including email, HTTP traffic both internally and externally to the Internet, voice over IP, FTP, and more, slow traffic is becoming a reality. Important traffic may even be dropped or slowed to an unusable speed. Web traffic delays can result in a loss of revenue for businesses. Traffic shaping attempts to normalize traffic peaks and bursts to prioritize certain flows over others. There's a physical limitation to the amount of data that can be buffered and to the length of time it can be buffered.

FortiGate devices provide Quality of Service (QoS) by applying bandwidth limits and prioritization. You can use traffic shaping to adjust how a FortiGate allocates resources to different types of traffic to improve performance and stability of latency sensitive or bandwidth intensive network applications.

Traffic shaping, or traffic management, controls the bandwidth available and sets the priority of traffic processed by the policy to control the volume of traffic for a specific period (bandwidth throttling) or rate the traffic is sent (rate limiting).

Traffic shaping attempts to normalize traffic peaks and bursts to prioritize certain flows over others. But there is a physical limitation to the amount of data which can be buffered and to the length of time. Once these thresholds are surpassed, frames and packets are dropped, and sessions are affected in other ways.

A basic traffic shaping approach is to prioritize certain traffic flows over other traffic whose potential loss is less disadvantageous. This means that you accept certain sacrifices in performance and stability on low-priority traffic, to increase or guarantee performance and stability on high-priority traffic.

If, for example, you're applying bandwidth limitations to certain flows, you must accept the fact that these sessions can be limited and therefore negatively impacted.

Note that traffic shaping is effective for normal IP traffic at normal traffic rates. Traffic shaping isn't effective during periods when traffic exceeds the capacity of the FortiGate. Because packets must be received by the FortiGate before they're subject to traffic shaping, if the FortiGate can't process all of the traffic it receives, then dropped packets, delays, and latency are likely to occur.

To ensure that traffic shaping is working at its best, make sure that the interface Ethernet statistics show no errors, collisions, or buffer overruns.

Accelerated interfaces (NPx network processors and CE) affect traffic shaping. For more information, see Hardware acceleration.

QoS

Quality of Service (QoS) is the capability to adjust some quality aspects of your overall network traffic. This can include such techniques as priority-based queuing and traffic policing. Because bandwidth is finite and because some types of traffic are slow, jitter or packet loss sensitive, bandwidth intensive, or operation critical, QoS can be a useful tool for optimizing the performance of various applications on your network.

Before implementing QoS, you should first identify the types of traffic that are important to your organization, the types of traffic that use high amounts of bandwidth, and the types of traffic that are sensitive to latency or packet loss.

For example, a company might want to guarantee sufficient bandwidth for revenue producing e-commerce traffic. They need to ensure that transactions can be completed and that clients don't experience service delays and interruptions. At the same time, the company may need to ensure low latency for voice over IP (VoIP) traffic used by sales and customer support, while traffic latency and bursts may be less critical to the success of other network applications such as long term, resumable file transfers. Many organizations discover that QoS is especially important for managing their voice and streaming multimedia traffic. These types of traffic can rapidly consume bandwidth and are sensitive to latency.

Discovering the needs and relative importance of each traffic type on your network helps you to design an appropriate overall approach, including how you to configure each available QoS component technique. Some organizations discover that they only need to configure bandwidth limits for some services. Other organizations determine that they need to fully configure interface and security policy bandwidth limits for all services, and prioritize queuing of critical services relative to traffic rate.

You can implement QoS on FortiGate devices using the following techniques:

Technique Description
Traffic policing Drops packets that don't conform to bandwidth limitations
Traffic shaping Ensures that the traffic may consume bandwidth at least at the guaranteed rate by assigning a greater priority queue if the guarantee isn't being met. Also ensures that the traffic can't consume bandwidth greater than the maximum at any given instance in time. Flows greater than the maximum rate are subject to traffic policing.
Queuing Transmits packets in the order of their assigned priority queue for that physical interface. All traffic in a higher priority traffic queue must be completely transmitted before traffic in lower priority queues is transmitted.

When you're deciding how to configure QoS techniques, it can be helpful to know when FortiGate devices employ each technique in the overall traffic processing flow, and the considerations that arise from those mechanisms.

Traffic policing

A FortiGate begins to process traffic as it arrives (ingress) and departs (egress) on an interface. In later phases of network processing, such as enforcing maximum bandwidth use on sessions handled by a security policy, if the current rate for the destination interface or traffic regulated by that security policy is too high, the FortiGate may drop the packet. Time spent on prior processing, such as web filtering, decryption, or IPS, is often wasted on packets that aren't forwarded. This applies to VLAN interfaces and physical interfaces.

You can prevent this wasted effort on ingress by configuring the FortiGate to preemptively drop excess packets when they're received at the source interface, before most other traffic processing is performed:

config system interface

edit <interface_name>

set inbandwidth <limit>

next

end

 

where <limit> is the bandwidth limit for incoming traffic, in kbps. Excess packets are dropped. If inbandwidth is 0, the rate isn't limited.

A similar CLI command is available that can be performed on egress:

config system interface

edit <interface_name>

set outbandwidth <limit>

next

end

 

As with ingress, setting the rate to 0 (zero) sets the rate to unlimited.

Rate limiting traffic accepted by the interface allows you to restrict incoming traffic to rates that, while no longer the full capacity of the interface, at the traffic shaping point in the processing are more likely to result in acceptable rates of outgoing traffic per destination interface or all security policies. This conserves FortiGate processing resources for those packets that are more likely to be viable completely to the point of egress.

Excessive traffic policing can degrade network performance rather than improve it. For more information about factors that affect traffic policing, see Important considerations.

NP6 interfaces on FortiGate devices don’t fully support bandwidth limits. When you set the outbandwidth setting on an NP6 interface, the FortiGate implements a lower bandwidth limit than the one that you configure. The inbandwidth setting has no effect on an NP6 interface, unless you disable NP offloading for the traffic on that interface.

Bandwidth guarantee, limit, and priority interactions

After packet acceptance, the FortiGate classifies traffic and may apply traffic policing at additional points during processing. It may also apply QoS techniques, such as prioritization and traffic shaping. Traffic shaping consists of a mixture of traffic policing to enforce bandwidth limits, and priority queue adjustment to assist packets in achieving the guaranteed rate.

If you have configured prioritization, the FortiGate prioritizes egressing packets by distributing them among FIFO (first in, first out) queues associated with each possible priority number. Each physical interface has six priority queues. Virtual interfaces don't have their own queues, and instead use the priority queues of the physical interface to which they are bound.

Each physical interface’s six queues are queue 0 to queue 5, where queue 0 is the highest priority queue. However, for reasons described below, you may observe that your traffic uses only a subset of those six queues. Some traffic may always use a certain queue number. Some queuing may vary by the packet rate or mixture of services. Some queue numbers may be used only by through traffic for which you have configured traffic shaping in the security policy that applies to that traffic session. For example:

  • Administrative access traffic will always use queue 0.
  • Traffic matching security policies without traffic shaping may use queue 0, queue 1, or queue 2. Which queue will be used depends on the priority value you have configured for packets with that Type of Service (ToS) bit value, if you have configured ToS-based priorities.
  • Traffic matching security policies with traffic shaping may use any queue. Which queue will be used depends on whether the packet rate is currently below the guaranteed bandwidth (queue 0), or above the guaranteed bandwidth. Packets at rates greater than the maximum bandwidth limit are dropped.
  • If the global tos-based-priority is low (3), the priority in a traffic shaper is medium (2) and a packet flows though a policy that refers to the traffic shaper, the packet will be assigned the priority defined by the traffic shaper, in this case medium (2).

Prioritization and traffic shaping behavior varies according to your configuration, the service types and traffic volumes, and whether the traffic is through traffic, or the traffic originates from or terminates at the FortiGate itself.

FortiGate traffic

Security policies don't apply to administrative access to the FortiGate through HTTPS or SSH, or IPsec tunnel negotiations, and therefore FortiGate devices don't apply traffic shaping. Such traffic also uses the highest priority queue, queue 0. In other words, packet priority = 0.

Exceptions to this rule include traffic types that are connections related to a session governed by a security policy. For example, if you enabled scanning by FortiGuard antivirus, traffic from the sender technically terminates at the FortiGate proxy that scans that traffic type. The FortiGate initiates a second connection that transmits scanned content to its destination. Because the second connection’s traffic is technically originating from the FortiGate proxy and therefore the FortiGate itself, it uses the highest priority queue, queue 0. However, this connection is logically associated with through traffic, and is therefore subject to possible bandwidth enforcement and guarantees in its governing security policy. In this way, it behaves partly like other through traffic.

Through traffic

For traffic passing through the FortiGate, the method a FortiGate uses to determine the priority queue varies by whether traffic shaping is enabled or not. Packets may or may not use a priority queue directly or indirectly derived from the Type of Service (ToS) bit — sometimes used instead with differentiated services — in the packet’s IP header.

If traffic shaping isn't applied to a security policy, the FortiGate doesn't limit or guarantee bandwidth, and traffic for that session uses the priority queue determined directly by matching the ToS bit in its header with the configured values:

config system global

set traffic-priority tos

set traffic-priority-level {high | low | medium}

end

 

or, if you have configured a priority specifically for that ToS bit value:

config system tos-based-priority

edit <id_int>

set tos [0-15]

set priority {high | low | medium}

end

 

where tos is the value of the ToS bit in the packet’s IP header, and high has a priority value of 0 and low is 2. Priority values configured in the second location will override the global ToS-based priority. In other words, packet priority = ToS-based priority.

For example, you might specify that packets with a ToS bit value of 2 should use queue 0, which is the highest priority queue:

config system tos-based-priority

edit 15

set tos 2

set priority high

next

end

 

If traffic shaping is applied to a security policy using a shared shaper, the FortiGate may subject packets to traffic policing or priority queue increases in an effort to meet bandwidth guarantees configured in the shaper.

For example, you might create a shared traffic shaper, where high has a priority value of 1 and low is 3, and <rate> is the bandwidth limit in kilobits per second:

config firewall shaper traffic-shaper

edit <shaper_name>

...

set priority {high | medium | low}

set maximum-bandwidth <rate>

set guaranteed-bandwidth <rate>

end

 

Note that it's also necessary to create a traffic shaping policy and set it to use the shared traffic shaper:

config firewall shaping-policy

edit <policy ID>

...

set srcaddr <source address>

set dstaddr <destination address>

set service <service name>

set dstintf <destination interface list>

set traffic-shaper <shaper_name>

end

 

The following diagram shows traffic queuing as the packet rate increases.

  • If the current packet rate is less than guaranteed bandwidth, packets use priority queue 0: packet priority = 0.
 
  • If the current packet rate is greater than guaranteed bandwidth but less than maximum bandwidth, the FortiGate assigns a priority queue by adding the numerical value of the security policy-based priority, where the value of High is 1, and Low is 3, with the numerical value of the ToS-based priority, where high has a priority value of 0 and low is 2. Because the two values are added, depending on the configured ToS-based priorities, packets in this category could use queues from queue 1 to queue 5. In other words: packet priority = ToS-based priority + security policy-based priority.
  • If you enabled traffic shaping in the security policy, and the security policy’s Traffic Priority is Low (value 3), and the priority normally applied to packets with that ToS bit is medium (value 1), then packets have a total packet priority of 4, and use priority queue 4.
  • If the current packet rate exceeds the maximum bandwidth, excess packets are dropped.

Calculation and regulation of packet rates

Packet rates specified for Maximum Bandwidth or Guaranteed Bandwidth are:

rate = amount / time

 

where rate is expressed in Kbps

Burst size at any given instant can't exceed the amount configured for maximum bandwidth. Packets that exceed this are dropped. Packets deduct from the amount of bandwidth available to subsequent packets and available bandwidth regenerates at a fixed rate. As a result, bandwidth available to a given packet may be less than the configured rate, down to a minimum of 0 Kbps.

Rate calculation and behavior can alternatively be described using the token bucket metaphor, where:

  • A traffic flow has an associated bucket, which represents burst size bounds, and is the size of your configured bandwidth limit.
  • The bucket receives tokens, which represent available bandwidth, at the fixed configured rate.
  • As time passes, tokens are added to the bucket, up to the capacity of the bucket. Excess tokens are discarded.
  • When a packet arrives, the packet must deduct bandwidth tokens from the bucket equal to its packet size in order to egress.
  • Packets can't egress if there are insufficient tokens to pay for its egress. These nonconforming packets are dropped.

Bursts aren't redistributed over a longer interval, so bursts are propagated rather than smoothed, although their peak size is limited.

Maximum burst size is the capacity of the bucket (the configured bandwidth limit). Actual size varies by the current number of tokens in the bucket, which may be less than bucket capacity, due to deductions from previous packets and the fixed rate at which tokens accumulate. A depleted bucket refills at the rate of your configured bandwidth limit. Bursts can't borrow tokens from other time intervals. This behavior is illustrated in the graph below.

Bursts and bandwidth limits over time

By limiting traffic peaks and token regeneration in this way, the available bandwidth at any given moment may be less than bucket capacity, but your limit on the total amount per time interval is ensured. Total bandwidth use during each interval of one second is, at most, the integral of your configured rate.

You may observe that external clients, such as FTP or BitTorrent clients, initially report rates between Maximum Bandwidth and twice that of Maximum Bandwidth, depending on the size of their initial burst. This is notably so when a connection is initiated following a period of no network activity. The apparent discrepancy in rates is caused by a difference in perspective when delimiting time intervals. A burst from the client may initially consume all tokens in the bucket, and before the end of one second, as the bucket regenerates, be allowed to consume almost another bucket’s worth of bandwidth. From the perspective of the client, this constitutes one time interval. From the perspective of the FortiGate, however, the bucket can't accumulate tokens while full. Therefore, the time interval for token regeneration begins after the initial burst, and doesn't contain the burst. These different points of reference result in an initial discrepancy equal to the size of the burst — the client’s rate contains it, but the FortiGate device’s rate doesn't. If the connection is sustained to its limit and time progresses over an increasing number of intervals, however, this discrepancy decreases in importance relative to the bandwidth total, and the client’s reported rate will eventually approach that of the configured rate limit for the FortiGate device.

For example, the Maximum Bandwidth might be 50 Kbps and there has been no network activity for one or more seconds. The bucket is full. A burst from an FTP client immediately consumes 50 kilobits. Because the bucket completely regenerates over one second, by the time almost another second has elapsed from the initial burst, traffic can consume another 49.999 kilobits, for a total of 99.999 kilobits between the two points in time. From the vantage point of an external FTP client regulated by this bandwidth limit, it therefore initially appears that the bandwidth limit is 99.999 Kbps, almost twice the configured limit of 50 Kbps. However, bucket capacity only regenerates at your configured rate of 50 Kbps, and so the connection can only consume a maximum of 50 kilobits during each second thereafter. The result is that as bandwidth consumption is averaged over an increasing number of time intervals, each of which are limited to 50 Kbps, the effects of the first interval’s doubled bandwidth size diminishes proportionately, and the client’s reported rate eventually approaches your configured rate limit. The following table shows the effects of a 50 Kbps limit on client reported rates.

Total size transferred (kilobits)Time (seconds)Rate reported by client (Kbps)
99.999 (50 + 49.999)199.999
149.999274.999
199.999366.666
249.999462.499
299.999559.998
349.999658.333
.........

Guaranteed Bandwidth can also be described using a token bucket metaphor. However, because this feature attempts to achieve or exceed a rate rather than limit it, the FortiGate doesn't discard non-conforming packets, as it does for Maximum Bandwidth. Instead, when the flow doesn't achieve the rate, the FortiGate increases the packets’ priority queue, in an effort to increase the rate.

Guaranteed and maximum bandwidth rates apply to the bidirectional total for all sessions controlled by the security policy. For example, an FTP connection may entail two separate connections for the data and control portion of the session. Some packets may be reply traffic rather than initiating traffic. All packets for both connections are counted when calculating the packet rate for comparison with the guaranteed and maximum bandwidth rate.

Important considerations

By implementing Quality of Service (QoS), you trade some performance and/or stability from traffic X by discarding packets or introducing latency in order to improve performance and stability of traffic Y. The best traffic shaping configuration for your network balances the needs of each traffic flow by considering not only the needs of your particular organization, but also the resiliency and other characteristics of each particular service.

For example, you may find that web browsing traffic is both more resistant to interruptions or latency and less business critical than UDP or VoIP traffic, and so you might implement less restrictive QoS measures on UDP or VoIP traffic than on HTTP traffic.

An appropriate QoS configuration also takes into account the physical limits of your network devices and the interactions of the QoS mechanisms described in Bandwidth guarantee, limit, and priority interactions.

You may choose to configure QoS differently based on the hardware limits of your network and FortiGate. Traffic shaping may be less beneficial in extremely high‑volume situations where traffic exceeds a network interface’s or your FortiGate model’s overall physical capacity. A FortiGate must have enough resources, such as memory and processing power, to process all traffic it receives, and to process it at the required rate. If it doesn't have this capacity, then dropped packets and increased latency are likely to occur. For example, if the total amount of memory available for queuing on a physical interface is frequently exceeded by your network’s typical packet rates, frames and packets must be dropped. In such a situation, you might choose to implement QoS using a higher model FortiGate, or to configure an incoming bandwidth limit on each interface.

Incorrect traffic shaping configurations can actually further degrade certain network flows, because excessive discarding of packets or increased latency beyond points that can be gracefully handled by that protocol can create additional overhead at upper layers of the network, which may be attempting to recover from these errors. For example, a configuration might be too restrictive on the bandwidth accepted by an interface, and may therefore drop too many packets, resulting in the inability to complete or maintain a SIP call.

To optimize traffic shaping performance, first ensure that the Ethernet statistics for the network interface are clean of errors, collisions, or buffer overruns. To check the interface, enter the following diagnose command to see the traffic statistics:

diagnose hardware deviceinfo nic <port_name>

 

If these aren't clean, adjust the FortiGate and settings of routers or other network devices that are connected to the FortiGate. For more information, see Troubleshooting traffic shaping.

Once Ethernet statistics are clean, you may want to use only some of the available FortiGate QoS techniques, or configure them differently, based on the nature of FortiGate QoS mechanisms described in Bandwidth guarantee, limit, and priority interactions.

Configuration considerations include:

  • For maximum bandwidth limits, ensuring that bandwidth limits at the source interface and security policy aren't too low, which can cause the FortiGate to discard an excessive number of packets.
  • For prioritization, considering the ratios of how packets are distributed between available queues, and which queue is used by which types of services. If you assign most packets to the same priority queue, it negates the effects of configuring prioritization. If you assign many high bandwidth services to high priority queues, lower priority queues may be starved for bandwidth and experience increased or indefinite latency. For example, you may want to prioritize a latency-sensitive service, such as SIP, over a bandwidth-intensive service such as FTP. Consider also that bandwidth guarantees can affect the queue distribution, assigning packets to queue 0 instead of their typical queue in high-volume situations.
  • You may or may not want to guarantee bandwidth, because it causes the FortiGate to assign packets to queue 0 if the guaranteed packet rate isn't currently being met. Comparing queuing behavior for lower-bandwidth and higher-bandwidth situations, this would mean that effects of prioritization only become visible as traffic volumes rise and exceed their guarantees. Because of this, you might want only some services to use bandwidth guarantees, to avoid the possibility that in high-volume situations all traffic uses the same queue, thereby negating the effects of configuring prioritization.
  • For prioritization, configure prioritization for all through traffic. You may want to configure prioritization by either ToS-based priority or security policy priority, but not both. This simplifies analysis and troubleshooting.

Traffic subject to both security policy and ToS-based priorities use a combined priority from both of those parts of the configuration, while traffic subject to only one of the prioritization methods will use only that priority. If you configure both methods, or if you configure either method for only a subset of your traffic, packets for which a combined priority applies will frequently receive a lower priority queue than packets for which you have only configured one priority method, or for which you have not configured prioritization.

For example, if both ToS-based priority and security policy priority both dictate that a packet should receive a medium priority, in the absence of bandwidth guarantees, a packet uses queue 3, while if only ToS-based priority is configured, the packet uses queue 1, and if only security policy-based priority is configured, the packet uses queue 2. If no prioritization is configured, the packet uses queue 0.

For example alternative QoS implementations that illustrate these considerations, see Examples.