python - High amount of Request to yield per parsed page blocks crawling -
i've crawl every subpage contains 300+ links need follow. crawl slows down after minute or / crawls 0 pages / min.
if rund crawl pages 10-50 links per page there same problem doesn't show.
i've configured 10 concurrent_requests , 10 processed items , reactor theradpool 400. means 3.000 yields per 10 processed items...
logging shows parse function takes 70+ seconds per page. logging shows time needed here comes yields (take 2 secounds per yield).
it seems scrapy waits until engine? or similar has done 1 task , ready process new yield request? adding requests scheduler don't take long seems me yield waiting other.
any hints whats tweak or whats going wrong?
is possible yield requests in bulk instead of yielding each on own? possible add them scheduler without yielding them?
some additional informations: - no difference if use scrapy-redis or disk-base scheduler. - because of rendering javascript download of crawled page can take 10 seconds. - autothrottle disabled - if give more cpu ressources doesn't speed things.
telnet-> est()
time()-engine.start_time : 676.0599975585938 engine.has_capacity() : false len(engine.downloader.active) : 7 engine.scraper.is_idle() : false engine.spider.name : onetwothree engine.spider_is_idle(engine.spider) : false engine.slot.closing : false len(engine.slot.inprogress) : 28 len(engine.slot.scheduler.dqs or []) : attributeerror (exception) len(engine.slot.scheduler.mqs) : attributeerror (exception) len(engine.scraper.slot.queue) : 0 len(engine.scraper.slot.active) : 21 engine.scraper.slot.active_size : 3878605 engine.scraper.slot.itemproc_size : 0 engine.scraper.slot.needs_backout() : false attributeerror seems come scrapy-redis plugin, without scrapy counts pages request in mgs. est if len(engine.downloader.active) goes 0
time()-engine.start_time : 7236.464096784592 engine.has_capacity() : false len(engine.downloader.active) : 0 engine.scraper.is_idle() : false engine.spider.name : onetwothree engine.spider_is_idle(engine.spider) : false engine.slot.closing : false len(engine.slot.inprogress) : 25 len(engine.slot.scheduler.dqs or []) : attributeerror (exception) len(engine.slot.scheduler.mqs) : attributeerror (exception) len(engine.scraper.slot.queue) : 0 len(engine.scraper.slot.active) : 25 engine.scraper.slot.active_size : 5357134 engine.scraper.slot.itemproc_size : 0 engine.scraper.slot.needs_backout() : true crawler code:
class robo2spider(spider): http_pass = none http_user = none dont_redirect = true start_urls = [] def __init__(self, *args, **kwargs): # ... config ... self.start_urls = self.custom_settings["test_urls"] # don't follow links in test mode def start_requests(self): url in self.start_urls: r = self.get_request(url) yield r def parse(self, response): # extraction , co... yield self.scrape_data(response) def scrape_data(self, response): start_time = time.time() # more extraction, build item extract_links = util.extract_links_from_response(response, self.query_pars_to_ignore) logging.info( "--- logging time 1: %s --- %s seconds ---" % (response.url, time.time() - start_time)) request_links = [] link in extract_links: if (not link["nofollow"]) , (l.get_output_value("crawl_meta_nofollow") != "nofollow"): r = self.get_request(link["url"]) request_links.append(r) yield request_links logging.info( "--- logging time 2 (takes 70 sec): %s --- %s seconds ---" % (response.url, time.time() - start_time)) yield l.load_item() def get_request(self, url, rit=none, splash_retry=none): # ... setting meta & co ... meta = {} splash_args = {} return splashrequest(url=url, callback=self.parse, meta=meta, args=splash_args, http_status_from_error_code=true, endpoint='execute', slot_policy=scrapy_splash.slotpolicy.scrapy_default) same result if way: link in extract_links:
if (not link["nofollow"]) , (l.get_output_value("crawl_meta_nofollow") != "nofollow"): r = self.get_viu_request(link["url"]) request_links.append(r) yield r
Comments
Post a Comment