locked
If-Modified-Since requests? RRS feed

  • Question

  • I was wondering how the msnbot makes conditional gets.  My site contains mostly dynamic pages, but also hosts a large section of product documentation (60,000+ pages each over 100k in size) that are rarely modified.  I added support for the If-Modified-Since header for those pages and now when I view my log files I see the Googlebot getting a 304 response to its requests, but the msnbot (and Yahoo-Slurp) still gets a 200 response with the page content.

    So, what information could the msnbot be looking for that googlebot doesn't require?  My only guess at this point is I need to include a Last-Modified header in the 200 response as well, and then that date will be used for future conditional gets (maybe Google just uses the date it first crawled a page if a Last-Modified header date isn't included?).

    Any help is appreciated.

    Thanks!
    -Dave
    Friday, September 19, 2008 5:14 PM

Answers

  • Hi,

     

    has this changed for you at all? If not, what have you tried (if anything)?

     

    Tuesday, October 28, 2008 3:16 PM

All replies

  • I'm going to add this to a support queue Jeremiah and I are starting. I will get back to you as soon as we get a solid answer (or maybe not so solid as the case may be...).

     

    Brett

     

    Wednesday, September 24, 2008 7:38 PM
  • Thats great, thanks!
    Wednesday, September 24, 2008 10:38 PM
  • Hi,

     

    has this changed for you at all? If not, what have you tried (if anything)?

     

    Tuesday, October 28, 2008 3:16 PM