Re: Whois lookups (was: 2010.10.04 NANOG50 day 1 morning notes posted)

[Mark Kosters <markk () arin net>]

On 10/4/10 4:58 PM, "David Conrad" <drc () virtualized org> wrote:

> On Oct 4, 2010, at 9:58 AM, John Curran wrote:
> > On Oct 4, 2010, at 1:25 PM, Seth Mattinen wrote:
> > > Or the new whois doesn't scale as well as the old one.
> >  New WHOIS scales much better than the old one; it would have
> >  extremely challenging to assemble enough equipment to handle
> >  the current query rate.  Look at the NANOG presentation slide
> >  for the exact query rate graph, but we're handling orders of
> >  magnitude more queries at present.
>
> Looking at the graph on your 3 slide, it looks like ARIN is getting around
> 3200 whois queries per second.  How much of that query load is a result of
> non-port 43 queries (that is, making use of the REST features in the new
> server)?  It looks like the exponentiation in query load started around the
> same time the Whois-RWS was deployed...

Traffic increases a lot over the course of a day and follows a diurnal
pattern. Right now we are seeing close to 7,000 queries per second during
the height of the day. The original Whois cluster that Whois-RWS replaced
could not serve more than 800 queries per second.

There were two spikes. The first was right after we deployed Whois-RWS. For
two months, we saw a consistent load maxing at 2400 queries per second. The
second spike happened on Sept 6. At that point, traffic jumped almost 3x to
the current max of 7,000 queries per second and has been pretty consistent
over the past month.

The patterns that we see are interesting. Most interesting is the spike
asking for ip addresses login servers for the likes of Facebook, AOL, and
Yahoo. This pattern emerged on Sept 6.  Various people have been looking at
this but no good explanation has yet been found. Your guess is good as mine
what the cause of this query growth.

Regards,
Mark