Distributed crawling of hyperlinked documents
US8812478B1 · kind B1 · utility
0Cited by
18References
20Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Sep 10, 2012 |
| Grant date | Aug 19, 2014 |
| Priority date | — |
| Expiry date | Sep 10, 2032 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Techniques for crawling hyperlinked documents are provided. Hyperlinked documents to be crawled are grouped by host and the host to be crawled next is selected according to a stall time of the host. The stall time can indicate the earliest time that the host should be crawled and the stall times can be a predetermined amount of time, vary by host and be adjusted according to actual retrieval times from the host.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.