Sitemap generating client for web crawler
US7801881B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 30, 2005 |
| Grant date | Sep 21, 2010 |
| Priority date | — |
| Expiry date | Aug 8, 2027 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/951
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods and systems for a sitemap generating client for web crawlers are described. The client accesses one or more sources of document information about the documents available on a website, such as the file system, access logs, or pre-made URL lists. Document information is extracted from the sources and one or more sitemaps are generated based on the extracted document information. A notification is transmitted to a remote computer, informing that the sitemap(s) are available for access and likely have been updated. If the remote computer is associated with a web crawler, the remote computer may access the sitemap(s) and use the sitemaps to schedule a crawl of documents included or available on the website.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.