Scheduler for search engine crawler
US8042112B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 30, 2004 |
| Grant date | Oct 18, 2011 |
| Priority date | — |
| Expiry date | Oct 5, 2029 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A search engine crawler includes a distributed set of schedulers that are associated with one or more segments of document identifiers (e.g., URLs) corresponding to documents on a network (e.g., WWW). Each scheduler handles the scheduling of document identifiers (for crawling) for a subset of the known document identifiers. Using a starting set of document identifiers, such as the document identifiers crawled (or scheduled for crawling) during the most recent completed crawl, the scheduler removes from the starting set those document identifiers that have been unreachable in each of the last X crawls. Other filtering mechanisms may also be used to filter out some of the document identifiers in the starting set. The resulting list of document identifiers is written to a scheduled output file for use in a next crawl cycle.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.