People Companies Advertise Archives Contact Us Jason Dowdell

Main > Archives > 2004 > October > RSS Feeds Utilize On-Demand Crawling But RSS Isn't a Space

Thursday, October 21, 2004

RSS Feeds Utilize On-Demand Crawling But RSS Isn't a Space

If you think RSS is a stand-alone sector on the internet and that it's going to become it's own industry then David Galbraith begs to differ. Here's the post from his blog.

David Galbraith: "I've heard three people refer to the 'RSS space' at Web 2.0. This is dangerous hype. RSS is not a space, its a description of a way to transport links with clean titles.

Advertising in RSS feeds will probably be worth $100 - $150 million within the next 18 months, and RSS readers will eventually be baked into all browsers as a fancy bookmarking feature - and that's it.
If people wanted to get excited about a piece of geekery that weblogs have helped drive then ping servers would be a better thing to look at. If you become the king of all ping servers then you have something that is a real threat to the core business of search engines.

When quantitative information such as price appears in RSS product feeds, then ping servers are hugely valuable and search engines based on crawling are fundamentally broken"

My Take on David's post:
Personally, one of the best things to come from RSS IMHO is the fact that unless you ping a blog search engine or aggregator, your blog won't get indexed. Basically, the blog engines are saying, we're not wasting bandwidth checking your rss feed unless you tell us to and if you make us come to your site an the feed hasn't changed then be warned.

So why don't Google and Yahoo use this same reasoning?
Well, it's their prerogative and mission to index the entire web, wether or not it's up to date. Man I love that, 'whether or not it's up to date'. The major engines know the majority of web sites don't have the ability or even care to notify the engines when their site is updated. Don't say that to a search engine marketer though, they'll say that's heresy. Wait, I think I am a search engine marketer, go figure. Anyway, the only search engines that it would make sense to use a crawl on demand feature would be engines and directories within specific vertical markets. Because vertical markets want to have the highest quality sites and most relevant unique content within their industry but also will want to save bandwidth due to cost constraints they'll love this model.

My prediction is that this 'crawl on demand' model will pervade only niche engines and directories but the big boys will continue to barrel down the road with their big v8 engines. Why, you ask? Because they can!

By Jason Dowdell at 11:52 PM | Comments (0)

Post a Comment

Subscribe to Marketing Shift PostsSubscribe to The MarketingShift Feed