Re: Cisco AnyConnect speed woes!

[Zachary McGibbon <zachary.mcgibbon+nanog () gmail com>]

We seem to have narrowed down the problem to our Cisco SCE packet shaper.
It seems to be misclassifying about 15-20% of the DTLS traffic into
encrypted bittorrent and since we have shaping rules in place to limit
torrent traffic, this was causing the issue.

To resolve the issue, we put the IP of our VPN ASA into a different package
on the SCE and did not apply any shaping rules to it.

We are still monitoring to be sure but we are quite confident this was the
issue.

So note to anyone out there using a shaper and has a DTLS VPN behind it,
check your classifications or whitelist your VPN box!

- Zachary

On Tue, Dec 9, 2014 at 7:39 PM, Zachary McGibbon <
zachary.mcgibbon+nanog () gmail com> wrote:
> Hi Roberto,
>
> - We have disabled the DTLS compression feature, this has been verified on
> the client side that compression says 'None'
> - We are not using the VPN load balancing feature, the two boxes are
> running in an active/standby configuration
> - Yes we are tunnelling all traffic however local lan access is available
> if the user checks the checkbox in their client
> - We are inspecting the following:
>   dns preset_dns_map, ftp, h323 h225, h323 ras, rsh, rtsp, esmtp, sqlnet,
> skinny, sunrpc, xdmcp, sip, netbios, tftp, ip-options, icmp
> - Jumbo frames are not configured
> - We are using the following encryption methods: AES128 and 2048 bit
> certificate
> - We are running ASA 9.2.2.8 on a 5545X
> - We are pushing the Anyconnect client version 3.1.05182
>
> Also, I should mention what I mean when we see slow speeds.  For example,
> my internet connection at home is a cable modem with 30mb down, 10mb up.  I
> have done a path mtu discovery to my VPN at work and it is 1500.  When I
> run an iperf to a server at the office without vpn I get about 28mb down,
> 9.5mb up.  When I connect to vpn, the iperf to the same server is about
> 1.2mb down, and 900k up.  This is way too slow!
>
> - Zachary
>
> On Tue, Dec 9, 2014 at 4:39 PM, Roberto <roberto () ipnetworks it> wrote:
>
> > > The big issue we are having is that many of our users are complaining
> > of low speed when connected to the VPN.
> > Please can you indicate more details ?
> >
> > Is it enabled on the ASA the "compression" feature ?
> > Is it enabled on the ASA the VPN Load Balancing feature ?
> > Are you using the AnyConnect FULL TUNNEL mode ?
> > Which are the inspection configured on the ASA for the "remote access"
> > clients ?
> > Have you configured the Jumbo MTU on the CISCO ASA interfaces ?
> > Which encryption are configured on the ASA (are you using Suite B
> > Algorithms) ?
> > Which version of ASA are you using ?
> > Which version of AnyConnect are you using ?
> >
> >
> > Note:
> > protocols such as L2TP/IPSec are not hardware accelerated -- the IPSec
> > portion of L2TP/IPSec is hardware-accelerated, but the L2TP portion is not.
> > Likewise, the SSL portions of SVC and WebVPN use hardware acceleration,
> > but the application layer protocols are done in software.
> >
> >
> > Best Regards,
> >
> > _________________________________
> > Roberto Taccon
> >
> > e-mail: roberto () ipnetworks it
> > mobile: +39 340 4751352
> > fax: +39 045 4850850
> > skype: roberto.taccon
> >
> > -----Messaggio originale-----
> > Da: NANOG [mailto:nanog-bounces () nanog org] Per conto di Zachary McGibbon
> > Inviato: martedì 9 dicembre 2014 21.18
> > A: Matthew Huff
> > Cc: NANOG
> > Oggetto: Re: Cisco AnyConnect speed woes!
> >
> > We are trying to use SSLVPN (udp 443) and results are really all over the
> > place.  Most of our complaints are users connecting on Teksavvy however we
> > haven't been able to reach anyone in their network team to find out if they
> > are doing any filtering or shaping on their side.
> >
> > We don't have a lot of traffic coming through Cogent, most of the users
> > are local here in Montreal on either Bell or Videotron and they traverse
> > through the QIX (www.qix.ca)
> >
> > On Tue, Dec 9, 2014 at 3:03 PM, Matthew Huff <mhuff () ox com> wrote:
> >
> > > Are you using SSLVpn or IPSEC with anyconnect? I have had more luck
> > > with performance with IPSEC than SSLVpn.
> > >
> > > Also, just because your ISP is saying that they aren't
> > > shaping/filtering, doesn't mean they aren't.
> > >
> > > We had major issues with users using AnyConnect when it was
> > > transversing Cogent. We were getting 5-10% packet loss (although the
> > > Cisco stats didn't show it), and it was choking on it.
> > >
> > > ----
> > > Matthew Huff             | 1 Manhattanville Rd
> > > Director of Operations   | Purchase, NY 10577
> > > OTA Management LLC       | Phone: 914-460-4039
> > > aim: matthewbhuff        | Fax:   914-694-5669
> > >
> > > -----Original Message-----
> > > From: NANOG [mailto:nanog-bounces () nanog org] On Behalf Of Zachary
> > > McGibbon
> > > Sent: Tuesday, December 9, 2014 2:42 PM
> > > To: NANOG
> > > Subject: Cisco AnyConnect speed woes!
> > >
> > > I'm looking for some input on a situation that has been plaguing our
> > > new AnyConnect VPN setup.  Any input would be valuable, we are at a
> > > loss for what the problem is.
> > >
> > > We recently upgraded our VPN from our old Cisco 3000 VPN concentrators
> > > running PPTP and we are now running a pair of Cisco 5545x ASAs in an
> > > HA active/standby pair.
> > >
> > > The big issue we are having is that many of our users are complaining
> > > of low speed when connected to the VPN.  We have done tons of
> > > troubleshooting with Cisco TAC and we still haven't found the root of
> > our problem.
> > > Some tests we have done:
> > >
> > >    - We have tested changing MTU values
> > >    - We have tried all combinations of encryption methods (SSL, TLS,
> > IPSec,
> > >    L2TP) with similar results
> > >    - We have switched our active/standby boxes
> > >    - We have tested on our spare 5545x box
> > >    - We connected our spare box directly to our ISP with another IP
> > address
> > >    - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and
> > our
> > >    IPS (HP Tipping Point)
> > >    - We have bypassed our Shaper and our IPS
> > >    - We made sure that traffic from the routers talking to our ASAs is
> > >    synchronous, OSPF was configured to load balance but this has been
> > > changed
> > >    by changing the costs on the links to the ASAs
> > >    - We have verified with our two ISPs that they are not doing any
> > kind of
> > >    filtering or shaping
> > >    - We have noticed that in some instances that if a user is on a low
> > >    speed connection that their VPN speed gets cut by about 1/3.  This
> > > doesn't
> > >    seem normal that the VPN would use this much overhead
> > >    - We do not have the issue when connecting to VPN directly on our own
> > >    network, only connections from the Internet
> > >
> > > If you have any ideas on what we could try net, please let me know!
> > >
> > > - Zachary