Adaptive crawl rates based on publication frequency
US8255385B1 · kind B1 · utility
1Cited by
10References
20Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Mar 22, 2011 |
| Grant date | Aug 28, 2012 |
| Priority date | — |
| Expiry date | Mar 22, 2031 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/958
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods and systems for determining an adaptive crawl rate for a Web crawler based on historical publication data from a Web source are provided. A frequency of publication of the Web source is determined over a specified period of time, and an adaptive crawl rate is calculated using the frequency of publication. The Web crawler is then deployed at the calculated adaptive crawl rate.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.