• HOME
• About
• Authors
• Resources

Google Expands Web Coverage

Written By Drupad Sil | April 18, 2008 | Share This | Print This Post

Post a comment »

Earlier this week, Google announced its practice of crawling forms on high-quality sites in order to expand its web coverage. From Google’s webmaster blog itself:

“Specifically, when we encounter a <FORM> element on a high-quality site, we might choose to do a small number of queries using the form. For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made. If we ascertain that the web page resulting from our query is valid, interesting, and includes content not in our index, we may include it in our index much as we would include any other web page.”

Google was also quick to assure webmasters that their crawl agent, Googlebot, could be prevented from crawling specific areas of sites, and would not crawl forms that require password inputs or use terms associated with personal information. The response from the online search community has been generally positive, as the change will allow Google to map slightly more of the deep web, the section of the Internet that is currently unsearchable. From Josh Catone at ReadWriteWeb:

“Last year [Google] ate through 100 exabytes of data, but there’s still a lot that it can’t get access to. Known as the deep web (or hidden web, or invisible web, etc.), it is estimated that the majority of online data is hidden safely from Google’s prying eyes — private intranets, unlinked pages, some non-textual content, and until today dynamic content returned via form input was all inaccessible to the search engine… it is estimated that the deep web is several orders of magnitude larger than the regular, public world wide web. While there is some content that Google will never — and should never — get its hands on, by crawling form results Google is now peering just a little bit deeper into the Internet.”

There are certainly SEO implications of this change. To get a better picture, we turned to our very own Mark Pilatowski, SEO Manager. Mark outlines some main points to be aware of:

“One of the benefits is that Google will be able to crawl and index additional pages that they may not have been able to find before. Since these pages require user input they will be more targeted for a variety of long tail keywords. This has the potential to provide a more personalized user experience for the user who lands on these pages.

It reinforces the need to write relevant and keyword rich content so Google can find important pages using keywords on the site. One of the most intriguing aspects is that Google is grabbing keywords they find on the site and placing them in text boxes to find more content. If they find unique and relevant content using those keywords they will be able to index those pages and theoretically rank those pages higher for those keywords in their own index.”

Of course, the ironic part of this whole change is that while Google is using bots to fill out forms on websites, they frown heavily on anyone who hits their servers with a bot.

From Google’s own Webmaster Help Center:

“Google’s Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries absorbs resources and includes using any software (such as WebPosition Gold™) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries.”

They even hate it when people exhibit bot-like behavior on their site.

While it seems like a case of “do as I say, not as I do”, let’s wait and see how much more of the web this allows Google to process and how it affects the page rankings of high-traffic sites. It’s certainly one small step for mankind into the unknowns of the Internet; the question is, in what direction?

Topics: Google, Search: Innovations, Technology |

3 Responses to “Google Expands Web Coverage”

1. Searchviews: Week in Review | Searchviews - Daily insights on Search Marketing, Social Media and SEO by Reprise Media. [ April 18th, 2008 at 12:20 pm ]

   […] Google Expands Web Coverage […]


2. Coverage » Google Expands Web Coverage [ April 19th, 2008 at 3:01 am ]

   […] Drupad Sil wrote an interesting post today on Google Expands Web CoverageHere’s a quick excerptEarlier this week, Google announced its practice of crawling forms on high-quality sites in order to expand its web coverage. From Google’s webmaster blog itself:. “Specifically, when we encounter a element on a high-quality site … […]


3. personalized world map [ May 6th, 2008 at 2:50 am ]

   […] blog itself: ???Specifically, when we encounter a &ltFORM&gt element on a high-quality site, wehttp://www.searchviews.com/index.php/archives/2008/04/google-expands-web-coverage.phpTCS Reports Record First Quarter 2008 Net Income, up 90% Sequentially Marketwire via Yahoo! Finance […]

Comments

Name (required)

Mail (will not be published) (required)

Website

Subscribe via RSS

Home | About | Authors | Resources

©2004-2008 Reprise Media, Inc. All Rights Reserved.