使用Java Jsoup的问题刮网站，网站不“滚动”

Question

我的问题是关于从特定网站上收集数据的可能性。目前，我的算法正在将HTML转换为文本，然后检查文件中包含的标记词，并求和标记的数量。

我的问题在于在刮网站的同时无法向下“滚动”。正如你所看到的，它正在检查一个twitter帐户上的标志数，但它仅限于50sh最新的tweets。我希望我说清楚了。

附注：我给了twitter一个例子，我不是在为twitter寻找特定的东西，而是更健壮的东西。

我将非常感谢任何提示。

健康问题+H1N1：0例

基础设施安全：2个实例

西南边境暴力事件：1起

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

import java.io.*;
import java.util.LinkedList;
import java.util.List;

public class Main {

static List<String> dhsAndOtherAgencies = new LinkedList<>();
static List<String> domesticSecurity = new LinkedList<>();
static List<String> hazmatNuclear = new LinkedList<>();
static List<String> healthConcern = new LinkedList<>();
static List<String> infrastructureSecurity = new LinkedList<>();
static List<String> southwestBorderViolence = new LinkedList<>();
static List<String> terrorism = new LinkedList<>();
static List<String> weatherDisasterEmergency = new LinkedList<>();
static List<String> cyberSecutiry = new LinkedList<>();
static String stream;



public static void main(String[] args) throws IOException {



    createLists();
    createStream();
    Raport raport = generateReport();
    System.out.println(raport);

}

static int flagStream(List<String> list, String stream){

    int counter = 0;
    for (String flag: list){
        if(stream.contains(flag)) {
            System.out.println(flag);
            counter++;
        }
    }

    return counter;
}

static Raport generateReport(){
    return new Raport(
            flagStream(dhsAndOtherAgencies,stream),
            flagStream(domesticSecurity,stream),
            flagStream(hazmatNuclear,stream),
            flagStream(healthConcern,stream),
            flagStream(infrastructureSecurity,stream),
            flagStream(southwestBorderViolence,stream),
            flagStream(terrorism,stream),
            flagStream(weatherDisasterEmergency,stream),
            flagStream(cyberSecutiry,stream)

    );

}
static void createStream() throws IOException {
    Document doc = Jsoup.connect("https://twitter.com/realDonaldTrump").userAgent("mozilla/17.0").get();
    stream = doc.text();
}
static void createLists() throws IOException {
    BufferedReader read = new BufferedReader(new FileReader("clearListAllCases.txt"));

    String input;

    int hashCounter = 0;

    while((input=read.readLine())!=null){
        if(input.charAt(0)=='#'){
            hashCounter++;
            continue;
        }
        switch (hashCounter){

            case 1:
                dhsAndOtherAgencies.add(input);
                break;
            case 2:
                domesticSecurity.add(input);
                break;
            case 3:
                hazmatNuclear.add(input);
                break;
            case 4:
                healthConcern.add(input);
                break;
            case 5:
                infrastructureSecurity.add(input);
                break;
            case 6:
                southwestBorderViolence.add(input);
                break;
            case 7:
                terrorism.add(input);
                break;
            case 8:
                weatherDisasterEmergency.add(input);
                break;
            case 9:
                cyberSecutiry.add(input);
                break;
        }
    }
}
}

 class Raport {

int a,b,c,d,e,f,g,h,i;
int totalFlags;

Raport(int a, int b, int c, int d, int e, int f, int g, int h, int i){
    this.a = a;
    this.b = b;
    this.c = c;
    this.d = d;
    this.e = e;
    this.f = f;
    this.g = g;
    this.h = h;
    this.i = i;
    totalFlags = a+b+c+d+e+f+g+h+i;
}



public String toString(){

    return "DHS & Other Agencies:\t\t\t"+a+" instances\n"+
            "Domestic security:\t\t\t\t"+b+" instances\n"+
            "HAZMAT & Nuclear:\t\t\t\t"+c+" instances\n"+
            "Health Concern + H1N1:\t\t\t"+d+" instances\n"+
            "Infrastructure Security:\t\t"+e+" instances\n"+
            "Southwest Border Violence:\t\t"+f+" instances\n"+
            "Terrorism:\t\t\t\t\t\t"+g+" instances\n"+
            "Weather/Disaster/Emergency:\t\t"+h+" instances\n"+
            "Cyber Security:\t\t\t\t\t"+i+" instances\n"+
            "TOTAL FLAGS:\t\t\t\t\t"+totalFlags+" instances";
}

}

Answer 1

我建议打开浏览器的developer选项卡，尝试找出网站使用哪个URL/endpoint为infinite scroll获取新项目，因为JSoup本身并不执行Javascript。然后可以使用JSoup调用endpoint并解析结果。

如果它不起作用，那么使用HtmlUnit或Selenium可能会更好，因为它们都是功能齐全的web浏览器API，您可以使用Java进行控制。