Recently had a problem where the old web guy for a site decided to restrict all search engines in the robots.txt.  This obviously isn’t very helpful when you are trying to grow a client’s business.

So I updated the robots.txt to allow everyone but still google webmaster was telling me that the robots.txt file was not allowing access.  I thought this was weird since I just changed it to allow all.  Turns out that google only downloads a new robots.txt file about once a day.  So even if you request to add the sitemap in google webmaster it will still use the old one.

A good way to check that your robots file is ok is to go to the tools section while in google webmaster and analyze robots.txt .  This will let you know about any problems with your robots.txt and show you the current version that google has cached.  You can also make temporary changes to the cached version to ensure what you are doing is what you would like to happen with spiders.  None of the changes made in that form will change the cached version, that can only be downloaded by google from the site. Modifying the cached version though and retesting it ensures that you have made the correct changes.

When first posting your feed to igoogle it takes a little while for it to pull in new posts that you write. Unless you are digg, engadget, or some other already well known and highly visited site and often updated site. When you first put your site feed on igoogle it can take anywhere from 4-72 hours and then as time progresses you should see faster updates to your igoogle rss widget for your feed. Just make sure that the url you are using for your feed is getting updated when you have new posts to your blog/cms/application.

• How to add an RSS feed to your igoogle home page
• igoogle