Re: ATM (was Re: too many routes)

[Richard Irving <rirving () onecall net>]

Sean M. Doran wrote:
> Cool, I love being talked down to by old guys.  It's
> refreshing and doesn't happen nearly frequently enough.

  I am not that old. And I did not intentionally talk down, I was tired
and grumpy, I am sorry if it came out that way. But, I would love to
respond to the technical points in your 
extremely "learn-ed" email. ( That is a genuine compliment, not sarcasm.
)

Now remember, I am argueing for ATM, not about the natural color of my
hair.

Or, the speed of light when chasing a south bound sparrow. ;)


> PDH is dead.

  Uhhh....
 
> End-to-end POTS is already dying.  Worldcom is making a
> big deal over relatively simple technology which shuffles
> fax traffic over the Internet. 

 Yeah, and they are pushing it around over ATM.

> However, there are neat
> plans for SS7 and neater plans for doing clever things
> with interpreting DTMF.

  Hey, did you ever notice that ATM is called broadband ISDN,
and regular ISDN is called narrow band.. And that SS7 is designed
to eventually intermesh these two technologies? This is still not
an argument: CON ATM.

  This is not intended to be flippant.... ?sp?

> In this model POTS and historical
> telco voice and data schemes become services rather than
> infrastructure.

   Yes, but the end user, in most cases, will still get POTS, as he knew
it. And don't argue for ISDN... It is far easier to push voice across a
clear 
channel T1 (Over ATM) using ISDN, than it is to map the RBS onto the
stuff. But I agree completely.

  
> The reason that you see "strange" or at least "unsmooth"
> load balancing along parallel paths is that except with
> fib and slow switching cisco had always forwarded packets
> towards the same destination out the same interface, and
> load balancing was performed by assigning (upon a cache
> fault) particular destinations to particular interfaces.
>
> (Leon was invented to blow away cached entries so that
> over time prefixes would slosh about from one interface to
> another as they were re-demand-filled into the cache.

  So ,you are saying, it is only Cisco who demonstrates this variability
?
Can anyone back that ?

> Points if you know who Leon and Viktor are.  Hint: they're
> both as dead as PDH.)

  This reminds me of Englands Queen. While "PDH may be dead",
it is, and will be, emulated for a long time to come.

  So, "The Queen is dead, long live the Queen"
 
  ;)

> With CEF these days you can load-balance on a per-packet
> basis.  This has the side effect that you cannot guarantee
> that packets will remain in sequence if the one way delay
> across the load-balanced paths is off by more than about
> half a packet transmission time.  However, you also get
> much more even link utilization and no ugly
> cache/uncache/recache at frequent intervals (which really
> sucks because unfortunately you have to push a packet
> through the slow path at every recache).
  I can't ever remember a sequencing issue over parallel ATM paths...
Perhaps I am just not experienced enough ;>

> So anyway, as I was saying, I'm ignorant about such
> matters.
  Hardly.

> > Audio sounds great with lots of variability
>
> So if you aren't holding a full-duplex human-to-human
> conversation you introduce delay on the receiver side
> proportional to something like the 95th percentile and
> throw away outliers.  If you're holding a full-duplex
> long-distance human-to-human conversation you can use POTS
> (which is dying but which will live on in emulation) and
> pay lots of money or you can use one of a number of rather
> clever VON member packages and pay alot less money but put
> up with little nagging problems.  For local or toll-free
> stuff, to expect better price-performance from an end-user
> perspective now would require taking enormous doses of
> reality-altering drugs.

   All is quiet in the BoardRoom, the hostile merger is about to occur.
All the Executives stand pensive, awaiting the last second instructions
from the CEO, in germany. 

  " And now, whatever you do, don't forget the *snap*, *crackle*, *pop*, 
    or it will cost us millions, Oops! gotta run, ta. (click)"

I am certainly not saying voice over IP is not viable... for the right
situations.

Matter of Fact, voice over IP, over ATM, is pretty solid ;)

> Soft real time things can be implemented across a wide
> variety of unpredictable media depending on the window
> available to service the real time events and the slope of
> the utility decay function.
>
> For instance, interactive voice and video have a number of
> milliseconds leeway before a human audience will notice
> lag.  Inducing a delay to avoid missing the end of the
> optimal window for receiving in-sequence frames or blobs
> of compressed voice data is wise engineering, particularly
> if the induced delay is adjusted to avoid it itself
> leading to loss of data utility.

  So , what do you think CDVT is all about ;)  (I know , you already
knew this,
  I just couldn't resist )I am still looking  for the CON ATM in this
point....


> > However, I have NEVER failed to get the bandwith
> > "promised" in our nets.
>
> Sure, the problem is with mixing TCP and other window-based
> congestion control schemes which rely on implicit feedback
> with a rate-based congestion control scheme, particularly
> when it relies on explicit feedback.   The problem is
> exacerbated when the former overlaps the latter, such that
> only a part of the path between transmitter and receiver
> is congestion controlled by the same rate-based explicit
> feedback mechanism.
>
> What happens is that in the presence of transient
> congestion unless timing is very tightly synchronized
> (Van Jacobson has some really entertaining rants about
> this) the "outer loop" will react by either hovering
> around the equivalent of the CIR or by filling the pipe
> until the rate based mechanism induces queue drops.
> In easily observable pathological cases there is a stair
> step or vacillation effect resembling an old TCP sawtooth
> pattern rather than the much nicer patterns you get from a
> modern TCP with FT/FR/1321 stamps/SACK.
>
> In other words your goodput suffers dramatically.


   Good point. However, FGCRA helps, graceful drops and all that...
And yes, it will be a while before the entire mechanism is completely
tied together.
 Oh, never mind, you make my point in the following. And remember, I
specifically noted that legacy equipment does not count. 

> > But, doesn't that same thing happen when you over-run the receiving
> > router ?????
>
> Yes, and with OFRV's older equipment the lack of decent
> buffering (where decent output buffering is, per port,
> roughly the bandwidth x delay product across the network)
> was obvious as bandwidth * delay products increased.
>
> With this now fixed in modern equipment and WRED
> available, the implicit feedback is not so much dropped
> packets as delayed ACKs, which leads to a much nicer
> subtractive slow-down by the transmitter, rather than a
> multiplicative backing off.

    BYTW: I was arguing for ATM, not just ABR. But, the key here is to
provide a reliable method whereby ATM can convey information in the same
pro-active strategy.
My original point is that in the ATM exchange environment, it is far
less likely that an irresponsible neighbor will interrupt your service.
This
has scaling capacity .... And also a side effect I really like, only the
ones who
are abusing their lines, drop packets. (Usually...)
 
> Finally another ABR demon is in the decay of the rate at
> which a VS is allowed to send traffic, which in the face
> of bursty traffic (as one tends to see with most TCP-based
> protocols) throttles goodput rather dramatically.   Having
> to wait an RTT before an RM cell returns tends to produce
> unfortunate effects, and the patch around this is to try
> to adjust the scr contract to some decent but low value
> and assure that there is enough buffering to allow a VS's
> burst to wait to be serviced and hope that this doesn't
> worsen the bursty pattern by bunching up alot of data
> until an RM returns allowing the queue to drain suddenly.

  Or, just don't overaggregate to the Nth degree. ;\

Patient: Doctor, Doctor, it hurts when I do this.....
Doctor: So, don't do that. That will be 50 dollars.  ;)

  However, you have a very good point regarding the effect of delay
awaiting the RM to indicate congestion. The solution does not exist,
yet....
ATM is not without design considerations.
*Nothing* is a substitute for good planning. It is easier to avoid
bottlenecks when
the top circuit is 622mbs, and you can run parallel paths.

> >    Ahhh.. We await the completion, and proper interaction of RM, ILMI,
> > and OAM.
> > These will, (and in some cases already DO), provide that information
> > back to the router/tag switch.
> > Now do they use it well ?????
> > That is a different story....
>
> The problem is that you need the source to slow
> transmission down, and the only mechanism to do that is to
> delay ACKs or induce packet drops.  Even translating
> FECN/BECN into source quench or a drop close to the source
> is unhelpful since the data in flight will already lead to
> feedback which will slow down the source.

 Agreed, it is not currently as pro-active as an ideal situation would
be. Weaknesses still exist providing layer 2 information to the layer 3
devices. The temporary fix is to provide large ingress and egress
buffers, throughout the path, and provide as much end-to-end feedback as
possible, preferably with the ability for all devices in a path to react
to that feedback.
OAM, RM, etc.
Then to do as you say, set a reasonable SCR, and hope you bought enough
buffers, and the CDVT is adequate to prevent loss.


> The congestion control schemes are essentially
> fundamentally incompatible.

   Lets just say "not optimal".

> > > Therefore, unless ABR
> > > is deliberately inducing queueing delays, there is no way
> > > your delay can be decreased when you send lots of traffic
> > > unless the ATM people have found a way to accelerate
> > > photons given enough pressure in the queues.
> >    More available bandwidth = quicker transmission.

   I still say lets switch to tachyons ;)

(Just for those who don't know already, I am not serious)


> >  6  core2-fddi3-0.san-francisco.yourtransit.net (-.174.56.2)  567 ms
> > 154 ms
> > 292 ms
> >
> > > > > > > > > > > > > > > >   Tell me this is a speed of light issue.
> > > > > > > > > > > > > > > >   From the FDDI to the HSSI on the same router.
>
> This has nothing to do with the router's switching or
> route lookup mechanism.  Router requirements allow routers
> to be selective in generating ICMP messages, and cisco's
> implementation on non-CEF routers will hand the task of
> generating ICMP time exceededs, port unreachables and echo
> replies to the main processor, which gets to the task as a
> low priority when it's good and ready.  If the processor
> is doing anything else at the time you get rather long
> delays in replies, and if it's busy enough to start doing
> SPD you get nothing.

  I have noted the conversations regarding this. However, I should note
that according to the data we collected, there is a direct correlation
between
the timings increasing, and the variability/latency/loss of packets
travelling the path.

 My argument is based on our charting of timings of this nature, and the
results from BoardWatch reviews. Our timing charts matched almost
perfectly to boardwatches timing of
major carriers. The lower the timings, and closer to minimal variability
from these timings, the better the
final score from boardwatch.

 Perhaps this relationship is merely demonstrating that, indeed, ICMP
timing is "retarded" during periods of high router utilization, due to
prioritizing of processes. However, the fact that this prioritization 
is being "kicked in" proves to be a good indicator that a router is
experiencing a serious load. And routers experiencing heavy loads, are
*usually* the cuplrit in packet delay/variance/loss. Not conclusive
enough for
the courts, but quite informational , none-the-less.



> >   Flow switching does a route determination once per flow, after that
> > the packets are switched down a predetermined path "The Flow". Hence the
> > term "flow switching". This reduces the variability of
> > the entire flow.
>
> Um, no it doesn't.  As with all demand-cached forwarding
> schemes you have to process a packet heavily when you have
> a cache miss.   Darren Kerr did some really neat things to
> make it less disgusting than previous demand-cached
> switching schemes emanating out of OFRV, particularly
> with respect to gleaning lots of useful information out of
> the side-effects of a hand-tuned fast path that was
> designed to account for all the header processing one
> could expect.

  I have NO excuse for simplifying, other than a lack of need to get
more detailed. I am now a little more versed in my intended audience....
(Postmortem)
However, the concept still holds true. 
The flow switching is "in essence" demonstrating the advantage of
minimizing interim processing. We have collected similar data, as well.
(Hand tuned) 
BTW: Everyone seem to have missed that flow switching ,
in its current incarnation, does not scale well. 
Sort of like ethernet, there is a critical mass point.
I was surprised no one attacked with that.. 
(not that it was important to the issue, or anything)

> Flow switching does magic matching of related packets to
> cache entries which describe the disposition of the packet
> that in previous caching schemes could only be determined
> by processing individual packets to see if they matched
> various access lists and the like.  It's principal neat
> feature is that less per-packet processing means more pps
> throughput.
>
> MPLS is conceptually related, btw.
>
> Flow switching does not improve queueing delays or speed
> up photons and electrons, however, nor does it worsen
> them, therefore the effect of flow switching on
> variability of normal traffic is nil.


  Hold it.... doesn't a better PPS throughput imply reduced que latency
?
Not the physical process of entering the que, or the physical process
of exiting the que, but the median time the data is "que homed".
(again, full latency of ingress to egress)
Kind of "by definition" ?

  The more packets that you can clear,  the less congested the pipe.
The less congested a pipe, lower the probability that a packet
will need to experience delay. Or, have I over simplified ?

 Or, is your operational word "normal traffic"? That would make sense.  


> Flow switching has mechanisms cleverer than Leon the
> Cleaner to delete entries from the cache and consequently
> there are much reduced odds of a cache fault during a
> long-lived flow that is constantly sending at least
> occasional traffic.   You may see this as reducing
> variability.  I see it as fixing a openly-admitted design flaw.

 Ok. But, it still works. Reduce the time spent processing the packet/or
cell, and things get smoother... Why should it be otherwise?
 

> >  PS MAC Layer Switching, and ATM switching are apples and oranges.
> > Although, one could be used to do the other.
> > (Told you Dorian)
>
> Huh?

  Sorry, A response to another users mail. They referenced a holy war in
the archives. I derived from that thread that it was an argument about
traditional LAN
switches, and their MAC based flow descisions -versus- Layer 3 Routing
descisions. I guess
technically speaking, it was related. But,it was not the immediate
issue.

  I guess in the picture of things I created flame bait.
  I claim naivette' (sp ?) to the NANOG archives .
  However, I found the discussion refreshing.

  It was a pleasure   ;)

      Richard