Cisco ASA NetFlow flow-export active-refresh interval Problems

Did you recently upgrade your Cisco ASA and run into flow-export active refresh-interval problems? If you were starting to appreciate the numerous NetFlow Security Event Logging (NSEL) enhancements available in the Cisco ASA 8.4(5) NetFlow export you may be left disappointed after upgrading the ASA to version 8.5(1), 8.6(1), 8.7(1), 9.0(1), or 9.1(1). What happened?

Apparently the bidirectional flows and the flow-export active refresh-interval command (1 minute timeouts) were somehow left out of the above later versions. If you are new to these features, why would you care?  Here’s why:

• Bidirectional NetFlow:  Most NetFlow or IPFIX implementations are unidirectional meaning TCP connections between two hosts results in two flows (i.e. A to B and B to A). This sounds sort of inefficient, but here’s why it works the way it does.  Flows are generally only metered inbound or ingress on interfaces and the responding flow is captured on a different interface. Hence two flows. Bidirectional flows should be implemented according to RFC 5103 where a single flow represents A to B and B to A. Obviously the size of the flow increases by almost double however, it can result in nearly half the volume of flows back to the high volume NetFlow collector.  The SonicWALL IPFIX configuration is the only vendor we have seen implement this according to RFC 5103. Bidirectional flows from the Cisco ASA NetFlow export are not RFC 5103 compliant and have generally led to confusion. Without Cisco ASA 8.4(5) NetFlow, the Cisco ASA exports a single flow record with a total of how much traffic was transferred between both hosts.  However, the Cisco ASA NetFlow exports do not indicate which how much of the total went in each direction.

• flow-export active refresh-interval: NetFlow exporters support a command to control how often flows are exported to the NetFlow collector.  For example, a Cisco Router uses the command “ip flow-cache timeout active”. Cisco ASA NetFlow uses “flow-export active-refresh interval” to control how often the flows are exported. However, the problem is that “flow-export active-refresh interval” only exists in Cisco ASA v8.4(5). This means that in other version of Cisco ASA NetFlow you have to wait for a conversation on the network to end before the flows are exported. This is a serious problem when looking at persistent tunnels as the Cisco ASA NetFlow is not exported until the tunnel is torn down. This results in large amounts of flow data to all be exported at once after the conversation has ended rather than exporting flow data each minute. Cisco reported that the Cisco ASA NetFlow flow-export active-refresh interval command will be added back in v9.1(2) but, it still isn’t perfect.

Templates exported by the ASA in NSEL

The Cisco ASA exports multiple templates which represent different types of flows.

• Extended: if the flow is torn down before the configured delay, the flow-create event is not sent; an extended flow teardown event is sent instead.
• Denied: flow was explicitly denied from being created in the first place. A Denied no XLATE event shows that the event was denied and no translation of the source and destination IP addresses and ports is done. This is typical when using NAT addresses.
• Flow Created: event is exported as soon as the flow is created
• Teardown: events indicate that an existing flow in the flow database of the appliance has ended. It could be due to “natural” causes (TCP: fin/fin-ack/ack, UDP: firewall times it out), or it could be a flow that has a problem detected midstream and the firewall shuts it off. The Teardown event will give you the total byte count (both inbound and outbound) for the entire flow in the octetTotalCounts field.
• Flow Update: For long lived flows, records are periodically sent with the delta byte counts since the last flow-update event.  Nothing is sent for the last minute the flow is active (I.e. bytes are lost). Also nothing is sent for short lived flows (I.e. no bytes are sent).

Keeping the 5 templates outlined above in mind.  The follow is the flow export behavior in version 9.1(2).

1. If only the flow-update template is exported and a flow is short lived, no information is sent on the flow.
2. If only the flow-teardown template is exported and the flow is short lived. At flow teardown, a flow-teardown record is sent containing the total byte count over the life of the flow.
3. If both the flow-update and flow-teardown templates are exported and the flow is short lived. At flow teardown, only a flow-teardown record is sent containing the total byte count over the life of the flow.
4. If only the flow-update template is exported and a flow is long lived, flow-update records are periodically sent with the delta byte counts since the last flow-update event.  At flow teardown nothing is sent and the sum of all delta byte counts may fall short of the actual number of bytes sent over the life of the flow.  In other words, the actual flow is under reported because the last update is never sent.
5. If only the flow-teardown is exported and the flow is long lived.  At flow teardown, a flow-teardown record is sent containing the total byte count over the life of the flow.
6. If both the flow-update and flow-teardown are exported and the flow is long lived. Flow-update records are periodically sent with the delta byte count since the last flow-update event.   A flow-teardown record is also sent containing the total byte count over the life of the flow.

In short, the active timeout issue is still not resolved and vendors must use the flow-teardown to report on 100% of the data.  This can result in spikes in the trends with utilization well over 100% in some intervals.  Because of the above behavior, if the NetFlow reporting solution uses both the flow-update and flow-teardown templates the reports will be over stated.  This is because the flow tear-down contains all the bytes that were incrementally reported by the flow-update.  Also, the last few bytes of a long live flow are reported by tear-down templates but, not the flow-update..  If only the flow-update is used for reporting, then short lived flows aren’t reported. Confused?
Cisco Engineers are aware of the above and we are hoping that they address it soon.

A bit more about bidirectional NetFlow. Although the Cisco ASA was the first firewall to export NetFlow, other firewall vendors also export it. The Dell SonicWALL and Palo Alto Networks’ firewalls also support bidirectional NetFlow or in the case of SonicWALL it’s actually IPFIX which is the proposed standard for NetFlow. If you don’t have one of these protecting your network, you could be looking at Fortinet firewall sFlow support. Barracuda firewall IPFIX support is also starting to surface.

So, Cisco ASA NetFlow may have been the first to emerge but, the others are following suit. And most of these NetFlow exporting firewall vendors include rich contextual details by exporting elements with information on user name, ACL violated, application name, MAC address, etc. Remember I wrote ‘most’ as with the Fortinet sFlow support – it isn’t possible to export these details plus, it samples packets so accuracy can be a problem.

4/6/2016 update

Cisco filed bug: CSCui20863 . It appears that they have added a “flow delete” record type in the Flow Update template export. I believe that was to address one of the above issues and should provide full accounting for a flow. The change below is the recommended workaround. This theoretically means that you can modify the service policy rules to include “Created, Denied, Updated” for logging but not “Tear Down” on order to get accurate statistics.

Here are the configuration differences from this change:

policy-map global_policy
class flow_export_class
  flow-export event-type all destination 10.1.4.66

policy-map global_policy
class flow_export_class
  flow-export event-type flow-create destination 10.1.4.66
  flow-export event-type flow-denied destination 10.1.4.66
  flow-export event-type flow-update destination 10.1.4.66

Still have questions? Contact Plixer.