Web crawler scheduler that utilizes sitemaps from websites
US7769742B1 · kind B1 · utility
48Cited by
11References
31Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Jun 30, 2005 |
| Grant date | Aug 3, 2010 |
| Priority date | — |
| Expiry date | Jul 12, 2026 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.