网页刮擦W/AGE验证

Question

你好，我想使用python 3.x和BeautifulSoup从一个带有年龄验证弹出式窗口的网站上刮取数据。如果不点击“是”来表示“你是否超过21岁”，我就无法获得底层的文本和图像。谢谢你的支持。

编辑：谢谢，通过一些注释的帮助，我看到我可以使用cookie，但不确定如何管理/存储/调用请求包中的cookie。

编辑4：取得了一些进展，更新的代码如下。我设法在XML代码中找到了元素，现在我只需要设法单击链接。

#
import time
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup

driver = webdriver.Firefox(executable_path=r'/Users/jeff/Documents/geckodriver') # Optional argument, if not specified will search path.
driver.get('https://www.shopharborside.com/oakland/#/shop/412');

url = 'https://www.shopharborside.com/oakland/#/shop/412'
driver.get(url)

#
driver.find_element_by_class_name('hhc_modal-body').click(Yes)

#wait.1.second
time.sleep(1)

pagesource = driver.page_source
soup = BeautifulSoup(pagesource)

#you.can.now.enjoy.soup
print(soup.prettify())

编辑新：再次卡住，这里是当前的代码。我似乎已经隔离了元素“MBTNYES”，但在运行代码时出现错误：elementClickInterceptedException：message：element在点（625,278.5500030517578)，因为另一个元素遮蔽了它

 import time
 import selenium
 from selenium import webdriver
 from selenium.webdriver.common.keys import Keys
 from selenium.webdriver.support.ui import WebDriverWait
 from bs4 import BeautifulSoup

 driver = webdriver.Firefox(executable_path=r'/Users/jeff/Documents/geckodriver') # Optional argument, if not specified will search path.
 driver.get('https://www.shopharborside.com/oakland/#/shop/412');

 url = 'https://www.shopharborside.com/oakland/#/shop/412'
 driver.get(url)

 #

 driver.find_element_by_id('myBtnYes').click()

 #wait.1.second
 time.sleep(1)

 pagesource = driver.page_source
 soup = BeautifulSoup(pagesource)

 #you.can.now.enjoy.soup
 print(soup.prettify())

Answer 1

如果您的目标是单击验证get to selenium:ps，install selenium&&get geckodriver（firefox）或chromedriver（chrome）

#Mossein~King(hi i'm here to help)
import time
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.firefox.options import Options
from BeautifulSoup import BeautifulSoup

#this.is.for.headless.This.will.save.you.a.bunch.of.research.time(Trust.me)
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options)

#for.graphical(you.need.gecko.driver.for.firefox)
# driver = webdriver.Firefox()

url = 'your-url'
driver.get(url)

#get.the.link.to.clicking
#exaple if<a class='MosseinKing'>
driver.find_element_by_xpath("//a[@class='MosseinKing']").click()

#wait.1.secong.in.case.of.transitions
time.sleep(1)

pagesource = driver.page_source
soup = BeautifulSoup(pagesource)

#you.can.now.enjoy.soup
print soup.prettify()