Re: FCIP issues with SAN replication

[Martin Visser <martinvisser99 () gmail com>]

A couple of things that can help you analyse this.

While it does appear that Wireshark has a go at decoding the FCP/FCIP data,
there probably are HP specific extensions not recognised. However as Bill
and Gerald have alluded to, this fundamentally is just TCP traffic. To look
at it at a TCP level, just go to the meu Capture:Enabled Protocols and turn
off FCP and FCIP.

What is curious there is that the traffic flow is either 10.244.249.32
*to* 10.244.249.32 or 10.244.249.31 *to* 10.244.249.31. While there are two
different MAC addresses (a Cisco and QLogic) this does seem a bit strange.
Is there possibly an IP configuration issue for the FCIP network?

As Bill has pointed out the "bytes in flight" stays very small, which is
indicative that TCP window scaling is not being turned on. (As we don't see
the intial SYN/SYN-ACK handshake we can't tell whether scaling was enabled).
With a 35ms RTT and 32K window you simply not ramp up to use the full pipe.
Bill has explained this well, but if you "google"  for bandwidth-delay
product you will understand this a little more. You really want to have a
window size of at least 4MB to start to see this link being used to its full
extent.

Note that at the TCP level you can also see there is no TCP retransmission
or lost segments so no indication of congestion there.

I suspect that there is some configuration or tuning things to be done on
your storage kit.

Regards, Martin

MartinVisser99 () gmail com


On Wed, Jun 30, 2010 at 2:08 AM, Bill Meier <wmeier () newsguy com> wrote:

> Chandler, Mel wrote:
> > Greetings all,
>  > <snip>
> > Now the problem, although we have one gigabit of bandwidth, they'll only
> > use about 13Mbps of it each, we've verified this with iperf.  Each
> > connection we'll only take 13Mbps of bandwidth, parallel tests show each
> > connection gets 13Mbps of bandwidth.  The HP engineer told us that at
> > > 5Mbps we get approximately 1.3Mbps of actually data, which means that
> > FCIP has 80% over head?  Can that be right?  The big huge problem is
> > that after running for several hours they'll eventually just die and
> > have to rebooted to start replicating again.  They're already on the
> > latest firmware (2.4.4.1).  The only error we get from the statistic
> > screen of the MPX's says they're getting TCP timeouts.
> >
> >
> >
> > I've performed captures on both sides' MPXs' and the errors I see in a
> > 60 sec sample are FCP malformed packets (~4300), duplicate ACK's (~41),
> > previous segment lost (~3), fast retransmission (~3).  When HP was
> > questioned about the FCP malformed packets they stated that they use a
> > proprietary protocol and that wireshark wouldn't be able to decode it.
> > I've since searched for this protocol but can find no references to it
> > anywhere.  The other errors seem so minor and few it would be hard to
> > believe that they're impacting the data stream that much if at all.
> >
> >
> >
> > I'll include a small sample of the captures, if it lets me.
>
> Taking a quick look at one of the captures (SNA) (and if I haven't
> missed anything) I see the following.
>
> The rate appears to be limited because of the "TCP window size" being
> used and the TCP "acknowledge time" for the connection.
>
> 1. It appears that the "ack time" is about 35 millisecs (interval
> between when a frame is sent and when an ack is received).
>
> 2. The 'TCP window size is 32768 bytes).
>
>
> If we assume "infinitely fast" transmission then the pattern is
> essentially:
>
> - send 32K bytes;
> - wait 35 ms for the acks
> (and so on)
>
> So: ~ 1/.035sec * 32KBytes can be sent per second
>  which is ~ 8Mbits per/sec
>
> If my analysis is correct I suggest you may want to consult with ??
> about the configuration of the TCP connection.
>
> You can see the pattern in the SNA capture by:
>
> 1. Filter on the TCP/FC connection (Statistics ! Conversations ! TCP)
> 2. Select the first frame
> 3. Do Statistics ! TCP Stream Graph ! Tine sequence Graph (tcptrace).
>
> ___________________________________________________________________________
> Sent via:    Wireshark-users mailing list <wireshark-users () wireshark org>
> Archives:    http://www.wireshark.org/lists/wireshark-users
> Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
>             mailto:wireshark-users-request () wireshark org
> ?subject=unsubscribe
___________________________________________________________________________
Sent via:    Wireshark-users mailing list <wireshark-users () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-users
Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
             mailto:wireshark-users-request () wireshark org?subject=unsubscribe