Koyaanis Qatsi/When google last crawled wikipedia

Difference (from prior minor revision) (no other diffs)

Changed: 1,16c1

! Keeping within the Google cache is a pain.   (previously) (I guess I forgot to track it down for July. D'oh!) Google has crawled wikipedia through sometime between June 1 3:44 a.m. and 7:52 a.m. It has William Shakespeare/How to upload Shakespeare to Wikipedia--a good cause but not REM. My previous comment about BokoNism? was wrong because that entry's first revision is actually in March. The previous Google update began around May 5 2001 at 11:13 a.m. wikiserver time and ended on May 8 between 5:01 a.m. and 7:59 a.m. A Google search for "wikipedia RecentChanges" showed the last cached edit to be on May 5 at 11:13 a.m.; I'm expecting this to be near the beginning of the crawling of the site as it is the first link on the home page which does not link to itself. Then I searched for "wikipedia" and various topics, comparing Google's cache with Wikipedia's. As of 2:38 p.m. May 28 (Wikipedia time), Google had the 5:01 a.m. May 8 revision of Milgram experiment but not the introduction of Saxifragales (which was not actually an "edit," as shown). I have to admit that I knew crawling a site was not instantaneous, but I'm surprised by how long it took for google to crawl all of Wikipedia. --KQ I am sure that Google could crawl the site so fast it would make the servers little heads swim. The robot exclusion rule limits the rate at which a site can politely be spidered, however, so a complete crawl takes a long time. Typically, a large scale spider will try to stay at or below a page a minute. Thus, it would take days to spider a moderately large site like wikipedia. -- TedDunning Thanks, Ted. That does make sense; it's a wise precaution for smaller sites. --KQ /Google test page

---