Re: blogs.msdn.com: RSS consumes too much bandwidth

[Mark Nottingham <mnot@xxxxxxxx>]

An alternative would be to only update the feeds once or twice a day;  
surely their content isn't that time-sensitive, and it would give them  
a higher proportion of (much smaller) 304's.

Off the cuff, an interesting optimisation might be to use ETags and  
Last-Modified times to spread the resulting load so that it wasn't  
clumped around the publish time. It would have to be pretty carefully  
engineered, though.

As syndication gets more popular, it will also become easier to justify  
the full cost of serving it, because it'll be seen as essential for  
doing business. It will also be more likely that companies like Akamai  
and Digital Island will come up with specialised services that make it  
more affordable to do so.

At any rate, there are a number of things we (and the market) can do to  
mitigate this before we call RSS/Atom "broken."


On Sep 8, 2004, at 10:43 AM, Bob Wyman wrote:

> 	Robert Scoble notes on his blog[1] that blogs.msdn.com has started
> to feed abbreviated entries in order to reduce their bandwidth costs.  
> Scoble
> notes: "It's not scalable when 10s of thousands of people start  
> subscribing
> to thousands of separate RSS feeds and start pulling down those feeds  
> every
> few minutes." He says: "Bandwidth usage was growing faster than MSDN's
> ability to pay for, or keep up with, the bandwidth. Terrabytes of  
> bandwidth
> were being used up by RSS." Also: "RSS is losing some of its  
> advantages.
> More and more sites are not providing full-text feeds."
> 	This is, of course, not just an issue for RSS. It is an issue for
> Atom as well. While polling for RSS/Atom files has been an acceptable
> solution to-date, as the popularity of RSS/Atom increases, we're going  
> to
> see more and more people with popular feeds get hit with serious  
> bandwidth
> bills. Scoble says that this problem isn't just one felt by Microsoft.  
> He
> reports, "I know of a major broadcaster that refuses to turn on RSS  
> feeds
> because of this issue too."
> 	My hope is that in the process of defining Atom, we'll be able to
> address the bandwidth issue better than it has been addressed in the  
> past.
> There a number of things we can consider. One of them is getting  
> people to
> use push instead of pull (ala: "Atom over XMPP"[2,3]). There must be  
> other
> approaches we can use as well...
> 	What can we do to ensure that Atom doesn't get swamped by bandwidth
> problems?
>
> 		bob wyman
>
>
>
> [1] http://radio.weblogs.com/0001011/2004/09/08.html#a8195
> [2] http://pubsub.com/developers.php
> [3]
> http://www.ietf.org/internet-drafts/draft-saintandre-atompub-notify 
> -01.txt

--
Mark Nottingham     http://www.mnot.net/