Weird MTU and TCP retranmission problem

[Sam Stickland <sam_ml () spacething org>]

I haven't seen anything like this before, so I'm hoping someone here could 
enlighten me.

We have a customer that has taken a single co-located server from us. They 
can download large files from this server to any machine, except the Mac 
OS X machines at the end of their shared leased line at their office 
premises. Windows and OS9 machines at this site can download these files 
fine, as can Mac OS X machines at the end of consumer ADSL lines 
(offsite).

Downloads just stall shortly after starting, which initially appears to an 
MTU problem. Lowering the MTU on the affect Mac OS X machines fails to 
solve the problem until the MTU is set to 100 (yes 100) bytes. Strange 
that the windows machines don't have this problem. These affected Mac OS X 
machines don't experience this problem if the exact same files are 
downloaded from a different server in the same datacentre, behind the same 
router.

An ethereal dump of a failed download shows the following:

  1.873772  Mac -> Server TCP [TCP Dup ACK 203#32] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598
  1.874145 Server -> Mac  HTTP Continuation
  1.885515  Mac -> Server TCP [TCP Dup ACK 203#33] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598
  1.885889 Server -> Mac  HTTP Continuation
  1.897384  Mac -> Server TCP [TCP Dup ACK 203#34] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598
  1.897758 Server -> Mac  HTTP Continuation
  1.909627  Mac -> Server TCP [TCP Dup ACK 203#35] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598
  1.921996  Mac -> Server TCP [TCP Dup ACK 203#36] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598
  1.933865  Mac -> Server TCP [TCP Dup ACK 203#37] 7798 > http [ACK] 
Seq=786 Ack=106053 Win=65535 Len=0 TSV=4064715650 TSER=2370598

which leads me unsure as to whether the server is failing to receive the 
ACKs (hence the ACK retransmission) or whether the Mac is failing to 
receive the next packet so is retransmitting what it believes to be a lost 
ACK for the last packet?

The server is an HP Proliant running Windows 2003, setup and installed by 
HP. It's running the built-in windows firewall (ICF), but the effects are 
the same if this is disabled.

Any suggestions of where to continue to look would be very much 
appreciated.

Sam