System and method for enabling website owners to manage crawl rate in a website indexing system
US8032518B2 · kind B2 · utility
6Cited by
21References
36Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Sep 4, 2009 |
| Grant date | Oct 4, 2011 |
| Priority date | — |
| Expiry date | Sep 4, 2029 |
Classification
- Technology area (CPC Y)Emerging Cross-Sectional Technologies
- CPC primaryY10S707/99933
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.