I have been working with a number of customers recently who are required to use sampled NetFlow because of vendor configuration rules. In just about every case, reporting accuracy from sampled NetFlow has been a major concern.
With more traffic, comes more NetFlow records; with more NetFlow records, comes higher CPU utilization. High volume flow exports can consume too much bandwidth when sent across the network, and/or drastically impact performance of the switch or router. Depending on how much traffic you are working with, sometimes balancing the need for flow exports against device efficiency and performance are necessary priorities.
As a result many high end network device vendors are either forcing, or optionally recommending, that you use a sampled NetFlow configuration for flow exports.
Why is reporting accuracy from sampled NetFlow such a concern?
Using flow sampling for network traffic monitoring can leave some holes in what is being reported.
While it is true that a sampling rate of 1 out of 100 packets may reduce the export of NetFlow data by as much as 50 percent. You have to keep in mind that when sampling, a NetFlow collector is only receiving a small percentage of the traffic and will not properly represent total throughput or traffic details.
Vendors supporting sFlow add to their flow exports by using interface counters in conjunction with the IP packet samples. Poll Counters export accurate general interface use statistics much like using SNMP. The traffic details seen in reports still fall short because we are only reporting on what got sampled, 1 in every X packet headers, not all of them.
What is the impact when using sampled flows for Network security and detecting advanced persistent threats?
It all comes back to the holes in what is being collected and reported.
Today, network traffic monitoring using flow technologies to monitor communication behaviors, maintaining baselines, and detecting advanced persistent threats is becoming more relevant.
However, when security professionals need to go back in time and view a communication pattern, they find that sampled flows seldom contain the flows that they want to investigate.
So what options do you have when it comes to sampled NetFlow Accuracy?
- Don’t use Sampling.
If device resources and vendor configuration permit, use a 1 to 1 sample.
- Offload the NetFlow function to a NetFlow Probe Appliance.
The probe will take in mirrored traffic streams and create NetFlow record exports. You eliminate the resource burden on the network device and move it to the probe. The probe in turn is giving you 100% NetFlow visibility. You can also gain insight network performance by metering latency, jitter, and packet loss. The probe also provides Detailed layer-7 analysis of your DNS traffic to immediately alert when malware uses DNS for information ex-filtration or command and control.
- If the vendor allows, change the flow aggregation method.
Traditional NetFlow aggregates flows based on a standard tuple – Source IP, Destination IP, Source port, Destination port, Protocol, and input and output interfaces. In other words, all these elements must match to aggregate or a new flow record is created.
By changing the flow aggregation method on the NetFlow cache table you can cut down flow volume, and still get 100% traffic accountability.
Take a look at the drop in flow volumes from the exact same traffic streams where the 2 different aggregation methods are deployed.
- Last option. Simply because I have no other option – Sample.
In most sampled NetFlow configurations the sample rate is globally applied across all of the monitored interfaces. We can use the sample rate to apply a multiplier across the details seen on interfaces and in the reports.
There are couple of things to keep in mind when reporting on sampled traffic details.
- How consistent is the traffic traversing the interface or device?
Can we assume that a packet sampled as part of an ICMP or SNMP request carries the same traffic weight as a sampled packet from a file download?
Simply applying the multiplier to the sampled ICMP flow would most likely overstate the amount of traffic seen in that particular host to host conversation.
- The other part of the equation is – what did get sampled?
In many cases conversations may not show up at all in report details because they did not get sampled. This is very much the case on high volume devices where minimum sample rates are in thousands.
So you are really only applying the sample multiplier to the sampled traffic, not all of the traffic.
Sampled flows are great for understanding trends. If you really need sampled NetFlow accuracy and traffic visibility, set the sample rate to 1 to 1. This way you will be collecting all of the flows.
Is sampled NetFlow your only configuration option? If you would like to explore other network traffic monitoring options, contact us.