学校项目中的网页抓取

Question

我正在尝试使用Selenium从页面中提取数据。我上周做了，但这周有些变化，现在它不再工作了。问题是“显示更多”按钮，或者你们可以在网站上看到的“Prikaži broj”。我有多个页面要刮，但让我们集中在一个。

代码是：

options = Options()
options.headless = True
driver = webdriver.Chrome('/Users/Nenad/chromedriver', options=options)
driver.get('https://www.nekretnine.rs/stambeni-objekti/stanovi/zvezdara-konjarnik-milica-rakica-57m2-milica-rakica/NkJXDiY2ugE/')
try:
    element = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(2) > button:nth-child(2)').click()
    sleep(randint(3, 5))
    home_phone = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(2) > span:nth-child(1)')
    condo_agency_cell_phones.append(home_phone.text)
except:
    condo_agency_cell_phones.append('NaN')
try:
    element = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(4) > button:nth-child(2)').click()
    sleep(randint(3, 5))
    cell_phone = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(4) > span:nth-child(1)')
    condo_agency_cell_phones.append(cell_phone.text)
except:
    condo_agency_cell_phones.append('NaN')
driver.close()

上周它和xpath一起工作，但现在不行了。我甚至找到了一个按钮，但它没有点击：

options = Options()
options.headless = False
driver = webdriver.Chrome('/Users/Nenad/chromedriver', options=options)
driver.get('https://www.nekretnine.rs/stambeni-objekti/stanovi/zvezdara-konjarnik-milica-rakica-57m2-milica-rakica/NkJXDiY2ugE/')
sleep(20)
try:
    element = driver.find_element_by_xpath("//button\[@type='button'\]").click()
    print(element.text)
except:
    print('NaN')

Answer 1

如果第一个答案不能解决你的问题，那么试试这个。导入了一些不同的库。在上面的代码中，“try:”返回未定义变量的错误，因为库未导入。

from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from time import sleep
options = Options()
options.headless = True
driver = webdriver.Chrome('/Users/Nenad/chromedriver', options=options)
# driver = webdriver.Firefox(executable_path=r'C:\\Py\\geckodriver.exe');

driver.get('https://www.nekretnine.rs/stambeni-objekti/stanovi/zvezdara-konjarnik-milica-rakica-57m2-milica-rakica/NkJXDiY2ugE/')
condo_agency_cell_phones = []
try:
    element = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(2) > button:nth-child(2)').click()
    # sleep(randint(3, 5))
    sleep(4)
    # home_phone1 = driver.find_element_by_xpath("html/body/div[11]/div[1]/div[2]/div[1]/div/div[2]/div[2]/div/div/form[1]/span")
    # condo_agency_cell_phones.append(home_phone1.text)
    home_phone = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(2) > span:nth-child(1)')
    print(home_phone.text)
    condo_agency_cell_phones.append(home_phone.text)
except:
    condo_agency_cell_phones.append('NaN')
try:
    element = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(4) > button:nth-child(2)').click()
    # sleep(randint(3, 5))
    sleep ( 4 )
    cell_phone = driver.find_element_by_css_selector('div.row:nth-child(2) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > form:nth-child(4) > span:nth-child(1)')
    condo_agency_cell_phones.append(cell_phone.text)
except:
    condo_agency_cell_phones.append('NaN')

print(condo_agency_cell_phones)
driver.close()

Answer 2

而不是xpath，尝试寻找由css选择器find_element_by_css_selector（按钮[type="按钮"]）