Screen scraping JavaScript and AJAX heavy pages with Selenium

Screen scraping libraries are popular and easy to use in a variety of languages (such as Symfony Crawler in PHP, Scrapy in Python etc). As these libraries are HTML parsers and not browsers, they cannot render JavaScript. As web pages get more sophisticated, JavaScript has become a core part of the user experience; therefore the tools we use …

Continue reading