如何在python中使用selenium和beautifulsoup从网站上抓取报纸文章？

Question

我正在尝试从报纸（纽约时间）收集日期，标题和内容。

我有日期和标题，但没有完整的文章。下面是我用来抓取日期和标题的代码。

import time
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

base = "https://www.nytimes.com"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
wait = WebDriverWait(browser, 10)
browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

while True:
    try:
        time.sleep(1)
    show_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))
        show_more.click()
    except Exception as e:
            print(e)
            break

soup = BeautifulSoup(browser.page_source,'lxml')
search_results = soup.find('ol', {'data-testid':'search-results'})

links = search_results.find_all('a')
for link in links:
    title = link.find('h4').text
    date = link.find_next('time').text
    print(date + ': '+ title)

print("Complete")

browser.quit()

除了日期，标题外，我还想抓全文。

Answer 1

有可能的。您必须获取该文章的每个链接并将其拉出。我在下面添加了它，还整理了一下文章，因为有多余的空格，当您将它们全部合并在一起时，文本中没有空格。

import time
import requests
from bs4 import BeautifulSoup
import json
import string
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

base = "https://www.nytimes.com"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
wait = WebDriverWait(browser, 10)
browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

while True:
    try:
        time.sleep(1)
        show_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))  
        show_more.click()
    except Exception as e:
            print(e)
            break

soup = BeautifulSoup(browser.page_source,'lxml')
search_results = soup.find('ol', {'data-testid':'search-results'})

links = search_results.find_all('a')
for link in links:
    link_url = link['href']

    title = link.find('h4').text
    date = link.find_next('time').text
    print(date + ': '+ title + '\n')

    response = requests.get(base + link_url)
    soup_link = BeautifulSoup(response.text, 'html.parser')
    scripts = soup_link.find_all('script')
    for script in scripts:
        if 'window.__preloadedData = ' in script.text:
            jsonStr = script.text
            jsonStr = jsonStr.split('window.__preloadedData = ')[-1]
            jsonStr = jsonStr.rsplit(';',1)[0]

            jsonData = json.loads(jsonStr)

            article = []
            for k, v in jsonData['initialState'].items():
                w=1
                try:
                    if v['__typename'] == 'TextInline':
                        article.append(v['text'])
                        #print (v['text'])
                except:
                    continue
            article = [ each.strip() for each in article ]
            article = ''.join([('' if c in string.punctuation else ' ')+c for c in article]).strip()
    print (article + '\n')

print("Complete")

browser.quit()

输出：

March 31: Bezos’ Security Consultant Accuses Saudis of Hacking the Amazon C.E.O.’s Phone

SEATTLE — Jeff Bezos’ security consultant accused the Saudi government of gaining unauthorized access to the Amazon chief executive’s phone, as part of an effort to harm the world’s richest man. In an opinion article in The Daily Beast on Saturday, Gavin de Becker, Mr. Bezos’ security chief, alleged the Saudis wanted to hurt Mr. Bezos because he owns The Washington Post. The Post has aggressively reported on the murder of Jamal Khashoggi, one of its columnists, who was killed last year in Turkey. United States officials have concluded Mr. Khashoggi, who was critical of Saudi leaders, was killed on the orders of the Saudi crown prince, Mohammed bin Salman. Mr. de Becker said he had turned over his findings about the Saudis and their role against Mr. Bezos to law enforcement. “Our investigators and several experts concluded with high confidence that the Saudis had access to Bezos’ phone, and gained private information,” Mr. de Becker wrote. Mr. de Becker’s claims about the Saudis are difficult to verify and raise many questions. Throughout his article, the security consultant was vague on details. He did not reveal direct evidence of his accusations and wrote that he would not speak again publicly on the issue. On Sunday, American Media Inc., the parent company of The National Enquirer, which first reported earlier this year that Mr. Bezos was having an extramarital affair, issued a statement denying “the false and unsubstantiated claims of Mr. de Becker.” Amazon did not immediately have a comment about Mr. Bezos. An official with the Saudi embassy in Washington declined to comment on Saturday. Mr. de Becker’s allegations are the latest twist in a bizarre situation that has embroiled the largest U.S. tabloid publisher and Mr. Bezos, who runs Amazon and owns The Post. The unusual set of circumstances began in January, when Mr. Bezos announced that he and his wife, MacKenzie Bezos, were divorcing. The next day, The Enquirer published an exposé revealing that Mr. Bezos was romantically involved with Lauren Sanchez, a former Los Angeles TV anchor. Then in February, Mr. Bezos published a post on Medium accusing AMI of “extortion and blackmail.” Mr. Bezos said AMI had threatened to publish graphic photographs of him, including a “below-the-belt selfie,” if he did not publicly affirm that The Enquirer’s reporting on his affair was not motivated by political concerns. He said AMI, which has had ties to the Saudis, was “apoplectic” about The Post’s reporting on the Saudis. Mr. Bezos added that he had asked Mr. de Becker, his longtime security consultant, to investigate who had leaked information and photos about him. Earlier this month, The New York Times reported that two people with direct knowledge of The Enquirer’s reporting said that everything the tabloid received on Mr. Bezos’ affair, including the “below-the-belt selfie,” came from a single source. The Wall Street Journal later reported that AMI had paid Ms. Sanchez’s brother, Michael Sanchez, $200,000 for the texts. In its statement on Sunday, AMI said directly that its source was Mr. Sanchez. “The fact of the matter is, it was Michael Sanchez who tipped the National Enquirer off to the affair on Sept. 10, 2018, and over the course of four months provided all of the materials for our investigation.” AMI added, “There was no involvement by any other third party whatsover.” Mr. de Becker on Saturday said that the effort against Mr. Bezos went beyond Mr. Sanchez and also involved the Saudis. Mr. de Becker pointed to an article published on Saturday by The New York Post in which Mr. Sanchez said The Enquirer “had seen text exchanges between the secret couple” before he was in touch with the tabloid on the matter. “Reality is complicated, and can’t always be boiled down to a simple narrative like ‘the brother did it,’” Mr. de Becker wrote. Mr. de Becker said his investigation included interviews with cybersecurity experts and “people who personally know the Saudi Crown Prince Mohammed bin Salman.” But he stopped short of saying what methods he believed the Saudis may have used to access Mr. Bezos’ personal information. He added that AMI had wanted him to make a public statement “saying that my investigation had concluded they hadn’t relied upon ‘any form of electronic eavesdropping or hacking in their news-gathering process,’” and that the tabloid’s story was not “instigated, dictated or influenced in any manner by external forces, political or otherwise.” Mr. de Becker said he told AMI in a recorded call that those claims were “not my truth.”

March 29: In Ukraine, Russia Tests a New Facebook Tactic in Election Tampering

Campaigning for Ukraine’s presidential election had just begun to heat up when the authorities announced they had thwarted a Russian plot to use Facebook to undermine the vote. Unlike the 2016 interference in the United States, which centered on fake Facebook pages created by Russians in faraway St. Petersburg, the operation in Ukraine this year had ...

...