System and method for enabling website owners to manage crawl rate in a website indexing system
US7599920B1 · kind B1 · utility
49Cited by
1References
36Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Oct 12, 2006 |
| Grant date | Oct 6, 2009 |
| Priority date | — |
| Expiry date | May 23, 2027 |
Classification
- Technology area (CPC Y)Emerging Cross-Sectional Technologies
- CPC primaryY10S707/99933
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.