System and method for enabling website owner to manage crawl rate in a website indexing system
US8458163B2 · kind B2 · utility
0Cited by
25References
39Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Oct 3, 2011 |
| Grant date | Jun 4, 2013 |
| Priority date | — |
| Expiry date | Oct 3, 2031 |
Classification
- Technology area (CPC Y)Emerging Cross-Sectional Technologies
- CPC primaryY10S707/99933
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.