Techniques for crawling dynamic web content
US7536389B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Feb 22, 2005 |
| Grant date | May 19, 2009 |
| Priority date | — |
| Expiry date | Jun 3, 2026 |
Classification
- Technology area (CPC Y)Emerging Cross-Sectional Technologies
- CPC primaryY10S707/99953
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
An automated form filler and script executor is integrated with a web browser engine, which is communicatively coupled to a web crawler, thereby enabling the crawler to identify dynamic web content based on submission of forms completed by the form filler. The crawler is capable of identifying web pages containing forms that require submission, and JavaScript code that requires execution, respectively, for requesting dynamic web content from a server. The crawler passes a representation of such web pages to the browser engine. The form filler systematically completes the form based on various combinations of search parameter values provided by the web page for requesting dynamic content. Request messages are constructed by the browser engine and passed to the crawler for submission to the server. The dynamic content, received by the crawler from the server in response to the request, can be indexed according to conventional search engine indexing techniques.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.