scrapy start_requests не входит в функцию обратного вызова

Question

Не знаю, почему функция обратного вызова (синтаксический анализ) не вызывается для start_requests urls.It завершается без ввода в функцию синтаксического анализа.

Это мой cbrspider.py файл

class CbrSpider(scrapy.Spider): 
name = "cbr" 
allowed_domains = ["careerbuilder.com"] 
start_urls = (
     'http://www.careerbuilder.com/browse/category/computer-and-mathematical', 
) 
    def start_requests(self): 
     for i in range(1,2): 
      yield Request("http://ip.42.pl/raw", callback=self.parse_init) 
     for i in range(1,2): 
      yield Request("http://www.careerbuilder.com/jobs-net-developer?page_number="+str(i)+"&sort=date_desc", callback=self.parse) 
     for i in range(1,3): 
      yield Request("http://www.careerbuilder.com/jobs-it-manager?page_number="+str(i)+"&sort=date_desc", callback=self.parse) 


    def parse_init(self, response): 
     self.ip = response.xpath('//body/p/text()').extract() 

    def parse(self, response): 
     print "enter parse function" 
     for sel in response.xpath('//*[@class="job-list"]'): 
      item = CareerbuilderItem() 
      item['ip'] = self.ip[0] 
      item['name'] = sel.xpath('//div//h2[@class="job-title"]/a/text()').extract()[0] 
      item['location'] = sel.xpath('//div[@class="columns small-12 medium-3 end"]//h4[@class="job-text"]/text()').extract()[0] 
     yield item 

Answer 1

Это (который в значительной степени код, кроме как от объекта Item вы выскабливание, потому что у меня не было его), кажется, правильно работать (смотрите вывод), но с python 2.7.9, Scrapy 1.0.5 и Twisted 16.0.0. Какой питон вы используете?

Скрипт для запуска:

from subprocess import call 

call(["scrapy", "crawl", "cbr"]) 

или

from scrapy.crawler import CrawlerProcess 
from scrapy.utils.project import get_project_settings 

process = CrawlerProcess(get_project_settings()) 
process.crawl('cbr') 
process.start() # the script will block here until the crawling is finished 

Код:

from scrapy import Spider, Request 


class CbrSpider(Spider): 
    name = "cbr" 
    allowed_domains = ["careerbuilder.com"] 
    start_urls = (
      'http://www.careerbuilder.com/browse/category/computer-and-mathematical', 
    ) 

    def start_requests(self): 
     for i in range(1,2): 
      yield Request("http://ip.42.pl/raw", callback=self.parse_init) 
     for i in range(1,2): 
      yield Request("http://www.careerbuilder.com/jobs-net-developer?page_number="+str(i)+"&sort=date_desc", callback=self.parse) 
     for i in range(1,3): 
      yield Request("http://www.careerbuilder.com/jobs-it-manager?page_number="+str(i)+"&sort=date_desc", callback=self.parse) 


    def parse_init(self, response): 
     self.ip = response.xpath('//body/p/text()').extract() 

    def parse(self, response): 
     print "enter parse function" 
     for sel in response.xpath('//*[@class="job-list"]'): 
      item = {} 
      item['ip'] = self.ip[0] 
      item['name'] = sel.xpath('//div//h2[@class="job-title"]/a/text()').extract()[0] 
      item['location'] = sel.xpath('//div[@class="columns small-12 medium-3 end"]//h4[@class="job-text"]/text()').extract()[0] 
     yield item 

Часть продукции:

2016-05-09 13:11:18 [scrapy] INFO: Scrapy 1.0.5 started (bot: crawl_hhgreg) 
2016-05-09 13:11:18 [scrapy] INFO: Optional features available: ssl, http11 
2016-05-09 13:11:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'crawl_hhgreg.spiders', 'SPIDER_MODULES': ['crawl_hhgreg.spiders'], 'BOT_NAME': 'crawl_hhgreg'} 
2016-05-09 13:11:18 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2016-05-09 13:11:18 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2016-05-09 13:11:18 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2016-05-09 13:11:18 [scrapy] INFO: Enabled item pipelines: JsonWriterPipeline 
2016-05-09 13:11:18 [scrapy] INFO: Spider opened 
2016-05-09 13:11:18 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-05-09 13:11:18 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2016-05-09 13:11:20 [scrapy] DEBUG: Crawled (200) <GET http://ip.42.pl/raw> (referer: None) 
2016-05-09 13:11:22 [scrapy] DEBUG: Crawled (200) <GET http://www.careerbuilder.com/jobs-net-developer?page_number=1&sort=date_desc> (referer: None) 
enter parse function 
2016-05-09 13:11:22 [scrapy] DEBUG: Scraped from <200 http://www.careerbuilder.com/jobs-net-developer?page_number=1&sort=date_desc> 
{'ip': u'62.38.254.183', 'name': u'Systems Developer (Treasury Management) - 6111 N River Rd', 'location': u'\nRosemont, IL\n'} 
2016-05-09 13:11:23 [scrapy] DEBUG: Crawled (200) <GET http://www.careerbuilder.com/jobs-it-manager?page_number=1&sort=date_desc> (referer: None) 
enter parse function 
2016-05-09 13:11:23 [scrapy] DEBUG: Scraped from <200 http://www.careerbuilder.com/jobs-it-manager?page_number=1&sort=date_desc> 
{'ip': u'62.38.254.183', 'name': u'Medical Technologist', 'location': u'\nHonolulu, HI\n'}