Re: An appeal for more bandwidth to the Internet Archive

[Jonah Edwards <jonah () archive org>]

Thank you Jared & Tim!  Fortunately the recent upgrade has gotten us 
some overhead in the short term -- we generally run very close to the 
edge, capacity-wise, in order to get the most out of what we have, so a 
sudden unexpected burst in demand pinned us for a bit there.  I expect 
more upgrades to be necessary in the near future, particularly on the 
egress bandwidth and long-haul datacenter interconnect fronts, and would 
love to talk off-list about help with peering/pipes in the Bay Area and 
100G gear especially.

To answer a couple sibling thread questions, traffic is pretty 
legitimate -- lots of users accessing lots of content.  We're fine with 
folks downloading in bulk and most who do use our torrent functionality 
to do so.  We have a broad view of regionality of visitors, but we very 
intentionally don't keep specific usage records (after all, we are a 
library, and we don't track our patrons -- this also precludes the use 
of most commercial CDNs).  See e.g. 
https://archive.org/services/docs/api/views.html#footnote-what-is-the-deal-with-privacy-protecting-hashes-instead-of-ip-numbers-in-our-logs 
for some info.

As far as colocation costs, we run our own datacenters in the Bay Area, 
where most of our on-site staff lives and where fully ambient cooling is 
possible year-round.  Our presence in 200 Paul is a terminus for our 
dark fiber and a place to make XCs -- actually only a single rack.

Thanks again for the all the appreciation and help -

--
Jonah Edwards
jonah () archive org
4157638676

On 2020-05-12 09:33, Tim Požár wrote:
> Jared...
>
> Thanks for sharing this.  I was the first Director of Operations from
> '96 to '98, at was was then Internet Archive/Alex.  I was the network
> architect back then got them their ASN and original address space.
> Folks may also know, I help start SFMIX with Matt Peterson.
>
> A bit more detail in this...  Some of this I got from Jonah Edwards
> who is the current Network Architect at IA.  Yes, the bottle neck was
> the line cards.  They have upgraded and that has certainly helped the
> bandwidth of late.
>
> Peering would be a big help for IA. At this point they have two 10Gb
> LAG interfaces that show up on SFMIX that was turned up last February.
> Looking at the last couple of weeks the 95th percentile on this 20Gb
> LAG is 3 Gb.  As they just turned up on SFMIX, they are just starting
> to get peers turned up there. Eyeball networks that show up on SFMIX
> are highly encouraged to start peering with them.  Alas, they are v4
> only at this point.
>
> Additionally, if folks do have some fat pipes that can donate
> bandwidth at 200 Paul, I am sure Jonah won't turn it down.
>
> Tim
>
> On 5/12/20 4:45 AM, Jared Brown wrote:
> > Hello all!
> >
> > Last week the Internet Archive upgraded their bandwidth 30% from 47 
> > Gbps to 62 Gbps. It was all gobbled up immediately. There's a lovely 
> > solid green graph showing how usage grows vertically as each interface 
> > comes online until it too is 100% saturated. Looking at the graph 
> > legend you can see that their usage for the past 24 hours averages 
> > 49.76G on their 50G of transport.
> >
> > To see the pretty pictures follow the below link:
> > https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/
> >
> > Relevant parts from the blog post:
> > "A year ago, usage was 30Gbits/sec. At the beginning of this year, we 
> > were at 40Gbits/sec, and we were handling it. ...
> >
> > Then Covid-19 hit and demand rocketed to 50Gbits/sec and overran our 
> > network infrastructure’s ability to handle it.  So much so, our 
> > network statistics probes had difficulty collecting data (hence the 
> > white spots in the graphs).
> >
> > We bought a second router with new line cards, and got it installed 
> > and running (and none of this is easy during a pandemic), and 
> > increased our capacity from 47Gbits/sec peak to 62Gbits/sec peak.   
> > And we are handling it better, but it is still consumed."
> >
> > It is obvious that the Internet Archive needs more bandwidth to power 
> > the Wayback machine and to fulfill its mission of being the Internet 
> > library and the historic archive of our times.
> >
> > The Internet Archive is present at Digital Realty SFO (200 Paul) and a 
> > member of the San Francisco Metropolitan Internet Exchange (SFMIX).
> > I appeal to all list members present or capable of getting to these 
> > facilities to peer with and/or donate bandwidth to the Internet 
> > Archive.
> > I appeal to all vendors and others with equipment that they can donate 
> > to the Internet Archive to contact them so that they can scale their 
> > services and sustain their growth.
> >
> > The Internet Archive is currently running 10G equipment. If you can 
> > help them gain 100G connectivity, 100G routing, 100G switching and/or 
> > 100G DWDM capabilities, please reach out to them. They have the 
> > infrastructure and dark fiber to transition to 100G, but lack the 
> > equipment. You can find the Internet Archive's contact information 
> > below or you can contact Jonah at the Archive Org directly either by 
> > email or via the contact information available on his Twitter profile 
> > @jonahedwards.
> >
> > You can also donate at https://archive.org/donate/
> > The Internet Archive is a 501(c)(3) non-profit. Donations are  
> > tax-deductible.
> >
> >
> > Contact information:
> > https://archive.org/about/contact.php
> >
> > Volunteering:
> > https://archive.org/about/volunteerpositions.php
> >
> >
> > Disclaimer: I am not affiliated with the Internet Archive. Nobody 
> > asked me to write this post. If something angers you about this post, 
> > be angry at me. I merely think that the Internet Archive is a good 
> > thing and deserves our support.
> >
> > Jared