Re: An appeal for more bandwidth to the Internet Archive

[colin johnston <colinj () gt86car org uk>]

Is the increased usage due to more users or more existing users having higher bandwidth at home to request faster ?
Would be interested if IPS configured firewall used to block out invalid traffic/spam traffic and if such traffic 
increased when back end network capacity increased ?
What countries are requesting the most data and does this analysis throw up questions as to why ?
Are there high network usage hitters which raise question as to why asking for so much data time and time again and is 
this valid traffic use ?

Colin


> On 12 May 2020, at 17:33, Tim Požár <pozar () lns com> wrote:
>
> Jared...
>
> Thanks for sharing this.  I was the first Director of Operations from '96 to '98, at was was then Internet 
> Archive/Alex.  I was the network architect back then got them their ASN and original address space. Folks may also 
> know, I help start SFMIX with Matt Peterson.
>
> A bit more detail in this...  Some of this I got from Jonah Edwards who is the current Network Architect at IA.  Yes, 
> the bottle neck was the line cards.  They have upgraded and that has certainly helped the bandwidth of late.
>
> Peering would be a big help for IA. At this point they have two 10Gb LAG interfaces that show up on SFMIX that was 
> turned up last February. Looking at the last couple of weeks the 95th percentile on this 20Gb LAG is 3 Gb.  As they 
> just turned up on SFMIX, they are just starting to get peers turned up there. Eyeball networks that show up on SFMIX 
> are highly encouraged to start peering with them.  Alas, they are v4 only at this point.
>
> Additionally, if folks do have some fat pipes that can donate bandwidth at 200 Paul, I am sure Jonah won't turn it 
> down.
>
> Tim
>
> On 5/12/20 4:45 AM, Jared Brown wrote:
> > Hello all!
> > Last week the Internet Archive upgraded their bandwidth 30% from 47 Gbps to 62 Gbps. It was all gobbled up 
> > immediately. There's a lovely solid green graph showing how usage grows vertically as each interface comes online 
> > until it too is 100% saturated. Looking at the graph legend you can see that their usage for the past 24 hours 
> > averages 49.76G on their 50G of transport.
> > To see the pretty pictures follow the below link:
> > https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/
> > Relevant parts from the blog post:
> > "A year ago, usage was 30Gbits/sec. At the beginning of this year, we were at 40Gbits/sec, and we were handling it. 
> > ...
> > Then Covid-19 hit and demand rocketed to 50Gbits/sec and overran our network infrastructure’s ability to handle it.  
> > So much so, our network statistics probes had difficulty collecting data (hence the white spots in the graphs).
> > We bought a second router with new line cards, and got it installed and running (and none of this is easy during a 
> > pandemic), and increased our capacity from 47Gbits/sec peak to 62Gbits/sec peak.   And we are handling it better, 
> > but it is still consumed."
> > It is obvious that the Internet Archive needs more bandwidth to power the Wayback machine and to fulfill its mission 
> > of being the Internet library and the historic archive of our times.
> > The Internet Archive is present at Digital Realty SFO (200 Paul) and a member of the San Francisco Metropolitan 
> > Internet Exchange (SFMIX).
> > I appeal to all list members present or capable of getting to these facilities to peer with and/or donate bandwidth 
> > to the Internet Archive.
> > I appeal to all vendors and others with equipment that they can donate to the Internet Archive to contact them so 
> > that they can scale their services and sustain their growth.
> > The Internet Archive is currently running 10G equipment. If you can help them gain 100G connectivity, 100G routing, 
> > 100G switching and/or 100G DWDM capabilities, please reach out to them. They have the infrastructure and dark fiber 
> > to transition to 100G, but lack the equipment. You can find the Internet Archive's contact information below or you 
> > can contact Jonah at the Archive Org directly either by email or via the contact information available on his 
> > Twitter profile @jonahedwards.
> > You can also donate at https://archive.org/donate/
> > The Internet Archive is a 501(c)(3) non-profit. Donations are  tax-deductible.
> > Contact information:
> > https://archive.org/about/contact.php
> > Volunteering:
> > https://archive.org/about/volunteerpositions.php
> > Disclaimer: I am not affiliated with the Internet Archive. Nobody asked me to write this post. If something angers 
> > you about this post, be angry at me. I merely think that the Internet Archive is a good thing and deserves our 
> > support.
> > Jared