Google webmaster robots.txt only updated every 24 hours

Recently had a problem where the old web guy for a site decided to restrict all search engines in the robots.txt.  This obviously isn’t very helpful when you are trying to grow a client’s business.

So I updated the robots.txt to allow everyone but still google webmaster was telling me that the robots.txt file was not allowing access.  I thought this was weird since I just changed it to allow all.  Turns out that google only downloads a new robots.txt file about once a day.  So even if you request to add the sitemap in google webmaster it will still use the old one.

A good way to check that your robots file is ok is to go to the tools section while in google webmaster and analyze robots.txt .  This will let you know about any problems with your robots.txt and show you the current version that google has cached.  You can also make temporary changes to the cached version to ensure what you are doing is what you would like to happen with spiders.  None of the changes made in that form will change the cached version, that can only be downloaded by google from the site. Modifying the cached version though and retesting it ensures that you have made the correct changes.