explicit waits in selenium and phantomjs with python -

i'm trying scrape data off of site, , many other "wines" on site, , using selenium js site. however, i'm finding code works , other times not return values though nothing changing.

i think should use explicit waits selenium overcome challenge, i'm not sure how integrate them, guidance on doing helpful!

my code is

def ct_content(url):     browser = webdriver.phantomjs()     browser.get(url)     wait = webdriverwait(driver, 10)     html = browser.page_source     html = lxml.html.fromstring(html)     try:         content = html.xpath('//a[starts-with(@href, "list.asp?table=list")]/text()')         browser.quit()         return content     except:         browser.quit()         return false 

thanks!

try use more specific xpath:

//ul[@class="twin_set_list"]//a/text() 

also there no need use lxml. try:

from selenium.webdriver.support.ui import webdriverwait wait selenium.webdriver.support import expected_conditions ec  data = [link.get_attribute('textcontent') link in wait(browser, 10).until(ec.presence_of_all_elements_located((by.xpath, '//ul[@class="twin_set_list"]//a')))]