Web crawler scheduler that utilizes sitemaps from websites
US8417686B2 · kind B2 · utility
2Cited by
23References
21Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Oct 11, 2011 |
| Grant date | Apr 9, 2013 |
| Priority date | — |
| Expiry date | Oct 11, 2031 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.