nanog mailing list archives
Re: Reducing Usenet Bandwidth
From: Joe St Sauver <JOE () OREGON UOREGON EDU>
Date: Sat, 09 Feb 2002 10:34:00 -0800 (PST)
Date: Fri, 08 Feb 2002 22:09:12 -0800 From: Stephen Stuart <stuart () tech org> Subject: Re: Reducing Usenet Bandwidth To: David Schwartz <davids () webmaster com> Cc: nanog () merit edu
[snip]
Some of those problems in that vein that appeared insurmountable ten years ago might have solutions in current "peer-to-peer" networking technologies (thus the off-hand comment about Napster).
What you're really asking for is a NNTP-over-FastTrack (Kazaa/Morpheus) feed object. This is sort of/really an odd idea, but if you think about it, all the required parts are pretty easy to articulate in skeleton form: 1) Each file dumped into FastTrack/Kazaa/Morpheus would get as its "filename" the article's message-ID. A site that wanted to retrieve an article would issue (via FastTrack/Kazaa/Morpheus) a search request for that message-ID, retrieve the article from a FastTrack/Kazaa/Morpheus server that offered that file, and then add it to the local news news server spool (and/or, optionally, a FastTrack/Kazaa/Morpheus shared directory on the local server). 2) How would servers know what article IDs to ask for? Well, Usenet overview data for each group could also be published by news servers via FastTrack/Kazaa/Morpheus with filenames of the format pathtag!hierarchy.subhierarchy.group!yy:mm:dd:hh:mm:ss The pathtag would be required because obviously different news servers will know about different articles, and you will want to disambiguate which server's overview data you want (not because there's any expectation that any news server will in fact provide public access to any articles, but simply because one might expect that N servers might all simultaneously offer comparable but non-identical overview data for any given group, and you might want to choose the view from server foo, or bar, or baz over other alternatives). The hierarchy.subhierarchy.group component of the overview file name simply allows folks to identify the overview data file they'd want. The date/time stamp (GMT) provides a means of selecting the most recently available overview data from a list of matching responses. Overview data would obviously need to be cryptographically signed with a key tied to the pathtag to avoid problems with people providing forged overview data (wouldn't want people to be fed lists of bogus/inappropriate lists of message ID's as a DOS/spam attack, right?). Overview files could be generated at some server-determined periodicity, and to avoid inspiring any sort of synchronization effect, I'll omit mentioning a value here except to say that I've got something in the double-digits-of- minutes in mind here. (Besides, if you're going to publish overview data for some tens of thousands of groups, it will take a while to walk that tree). 3) Client servers interested in retrieving articles via FastTrack/Kazaa/Morpheus could routinely retrieve overview records for groups of interest, retrieving all/some of the articles as a "prefetch" for popular groups routinely read by their users, or the server could wait until articles have been requested, then retrieve an overview file, then retrieve articles. Alternatively, individual users with "News over FastTrack/Kazaa/Morpheus" clients could be retrieving articles directly from Kazaa, removing any need for an intervening news server whatsoever (although hopefully a local "regular" news server would always be faster than a distributed FastTrack/Morpheus/Kazaa-based news spool). Regards, Joe P.S. Needless to say, this would really change the FastTrack/Kazaa/Morpheus content base, and given the volumes Usenet is currently transferring, would be sort of an interesting experiment if we're talking about a full feed...
Current thread:
- Re: Reducing Usenet Bandwidth, (continued)
- Re: Reducing Usenet Bandwidth Joe St Sauver (Feb 09)
- Re: Reducing Usenet Bandwidth Stephen Stuart (Feb 09)
- Re: Reducing Usenet Bandwidth Stephen Griffin (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Griffin (Feb 11)
- RE: Reducing Usenet Bandwidth Deepak Jain (Feb 11)
- RE: Reducing Usenet Bandwidth Vadim Antonov (Feb 11)
- Re: Reducing Usenet Bandwidth Jonas M Luster (Feb 11)
- Re: Reducing Usenet Bandwidth Paul Vixie (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Stuart (Feb 11)
- Re: Reducing Usenet Bandwidth Jonas M Luster (Feb 11)
- Re: Reducing Usenet Bandwidth Joe St Sauver (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Sprunk (Feb 11)
- Re: Reducing Usenet Bandwidth David Charlap (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Griffin (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Stuart (Feb 11)
- Re: Reducing Usenet Bandwidth David Schwartz (Feb 11)
- Re: Reducing Usenet Bandwidth Stephen Stuart (Feb 12)
- Re: Reducing Usenet Bandwidth David Schwartz (Feb 12)
- Re: Reducing Usenet Bandwidth Stephen Stuart (Feb 12)
- Re: Reducing Usenet Bandwidth David Schwartz (Feb 11)
- Re: Reducing Usenet Bandwidth Joe St Sauver (Feb 09)