Scheduling resource crawls
US8868541B2 · kind B2 · utility
32Cited by
12References
17Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Jan 21, 2011 |
| Grant date | Oct 21, 2014 |
| Priority date | — |
| Expiry date | Feb 3, 2032 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scheduling resource crawls. In one aspect, a framework is provided for scheduling resource crawls such that a crawl scheduler determines the health of a document, i.e., whether it can be crawled, the popularity of the document, and the frequency of “interesting,” i.e., substantive, content changes, and based on this information, estimates an appropriate crawl interval for each web resource to improve crawl resource utilization.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.