For those who are surprised by the bandwidth problems that the polling of RSS feeds is causing the MS folks, let me say it once more:
UNCONTROLLED POLLING OF RESOURCES DOES NOT SCALE!!
20 years of developing distributed systems have thought us this, but the RSS advocates appear deaf for these issues. The simple approach that works well with small feeds and limited number of subscribers does not work with large feeds and many subscribers.
Whatever duct tape you put on this problem (e.g. make the subscribers poll less frequently) will get ripped off again when the growth to the next order of magnitude will happen. And if atom/rss is indeed such a great thing, this growth is indeed something that will happen.
There are simple, temporary solutions that will help big (group) feeds, such as using the 'changed-since' information from the client to serve a feed with only items that have changed since that time. But this will not solve the real problem.
For these highly popular (big) feeds, there is a range of solutions available. Many of them revolve around having multiple sources serve the feed, whether that is a cache (yes make your feed cacheable, that will provide automatic throttling), a neighbor (through collaborative techniques), through service aggregators (such as pubsub.com or bloglines), etc.
I have said this before, and I hope it the ms experience will trigger people who are responsible for large sites to start working on alternative architectures.
Posted by Werner Vogels at September 10, 2004 03:24 PMSince RSS/Atom were mostly developed by part-time volunteers, I wonder if they can afford to develop anything scalable. Of course, it goes without saying that anything scalable will be rejected by the "really simple" crowd. The question is how much leeway there is -- what level of expertise can we expect from an aggregator/publisher developer?
Posted by: Wes Felter on September 10, 2004 08:00 PM> Since RSS/Atom were mostly developed by part-time volunteers,
So was the World Wide Web.
> I wonder if they can afford to develop anything scalable.
Volunteer != idiot.
> Of course, it goes without saying that anything scalable will be rejected by the "really simple" crowd.
Pro-active notifications systems are already in production and built by part-time volunteers. Look at what TrackBack 'ping' really can do. They are event notification over HTTP - notifications on the Web. And check out http://www.mod-pubsub.org/ (specifically http://www.mod-pubsub.org/kn_apps/knownews/?/what/syndic8.com/news/items if you have IE and scripting turned on)
> The question is how much leeway there is -- what level of expertise can we expect from an aggregator/publisher developer?
The developers are going to follow standards, and if those standards are built within a framework that lends itself to scalable approaches, then it will evolve to become scalable. Or the product will die.
More on the issue today on CNet News:
http://news.com.com/Microsoft+flip-flop+may+signal+blog+clog/2100-1032_3-5368454.html
