We have improved our Web crawler to be able to deal with websites up to 5000 pages with rock solid performance. This is an entirely new crawling system that runs on a separate, scalable cloud hosting environment for maximum reliability.
There will be further improvements to the crawler in time, including the ability to crawl and scrape content from Javascript based websites.
To leave a comment, please authenticate.
+ 11
