python - srapy crawl mulitlayer append data in a list and yield this item -


the web site structure i'm trying parse using scrapy following:

structure

i'd extracted data have format:

[{     "project":{"projectname":"project1"},     "samples":["sample1's_content","sample2's_content","sample3's_content"]  },  {     "project":{"projectname":"project2"},     "samples":["sample1's_content","samples2's_content","sample3's_content"] }]

i tried this:

from item import item  class spider(scrapy.scrapy):     name = spider      def start_request(self):         url = "the main page's url"         yield scrapy.request(url=url, callback=self.parseprojectlist)      def parseprojectlist(self, response):         url in selector(project_list)             yield scrapy.request(url=url, callback=self.parseproject)      def parseproject(self, response):         #scrap data         myitem = item()         myitem['samples']=[]         myitem['project']={'projectname':projectname,...}         yield scrapy.request(url=samplelistpage, callback=self.parsesamplelistpage,meta={'myitem':myitem})      def parsesamplelistpage(self, response):         url in selector(sample_list)             yield scrapy.request(url=url, callback=self.parsesample,meta={'myitem':'myitem'})      def parsesample(self, response):         #parse sample data         response.meta['myitem'].append(sample_data)

i tried put yield response.meta['myitem'] @ parsesamplelistpage 

def parsesamplelistpage(self, response):         url in selector(sample_list)             yield scrapy.request(url=url, callback=self.parsesample,meta={'myitem':'myitem'})     yield response.meta['myitem']

and yield response.meta['myitem'] in parsesample

def parsesample(self, response):     #parse sample data     response.meta['myitem'].append(sample_data)     yield response.meta['myitem']

both solutions failed.

the first 1 yields empty "samples" fields. second 1 creates multiple data same project this:

[   {     "project": {       "projectname": "project2"     },     "samples": [       "sample1's_content"     ]   },   {     "project": {       "projectname": "project2"     },     "samples": [       "sample1's_content",       "sample2's_content"     ]   },   {     "project": {       "projectname": "project2"     },     "samples": [   "sample1's_content",   "sample2's_content",   "sample3's_content" ]   } ]

wonder there way deal problem?


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more