scrapyd 接口部分

前言

scrapyd接口一共有10个

---

一、daemonstatus.json

检查节点服务的状态，支持GET方式

curl http://localhost:6800/daemonstatus.json

GET
http://127.0.0.1:6800/daemonstatus.json

{
    "node_name": "***",
    "status": "ok",
    "pending": 0,
    "running": 2,
    "finished": 100
}

二、listprojects.json

查询发布到scrapy服务中的projects，支持GET查询

curl http://localhost:6800/listprojects.json

GET
http://127.0.0.1:6800/listprojects.json

{
    "node_name": "***",
    "status": "ok",
    "projects": [
        "weibo"
    ]
}

三、listversions.json

查看project下所有的版本，支持GET查询

curl http://localhost:6800/listprojects.json

GET
curl http://localhost:6800/listversions.json?project=weibo

{
    "node_name": "***",
    "status": "ok",
    "versions": [
        "2022-03-10T15_37_20",
        "2022-03-10T15_44_57",
        "2022-03-10T15_54_19"
    ]
}

四、listspiders.json

查询 project,_version下所有的spiders ,project必须指定 _version可有可无，支持GET查询

curl http://localhost:6800/listspiders.json?project=weibo

GET
http://127.0.0.1:6800/listspiders.json?project=weibo&_version=2022-03-10T15_37_20

{
    "node_name": "***",
    "status": "ok",
    "spiders": [
        "weibo",
        "weibo_file"
    ]
}

五、listjobs.json

查询project下所有的jobs，支持GET查询

curl http://localhost:6800/listjobs.json?project=weibo

GET
http://127.0.0.1:6800/listjobs.json?project=weibo

{
    "node_name": "***",
    "status": "ok",
    "pending": [],
    "running": [
        {
            "id": "task_2_2022-03-23T16_51_00",
            "spider": "weibo_file",
            "pid": 18988,
            "start_time": "2022-03-23 16:51:03.207089"
        }
    ],
    "finished": [
        {
            "id": "task_2_2022-03-23T13_57_00",
            "spider": "weibo_file",
            "start_time": "2022-03-23 13:57:03.204365",
            "end_time": "2022-03-23 13:58:24.336088"
        }
    ]
}

六、delversion.json

删除指定project,version的project，支持POST提交方式

curl http://localhost:6800/delversion.json -d project=myproject -d version=r99

POST 
http://127.0.0.1:6800/delversion.json
{"project":"weibo","version":"2022-03-10T15_37_20"}

{
    "node_name": "***",
    "status": "ok"
}

七、addversion.json

添加新版本的egg，支持POST提交方式

curl http://localhost:6800/addversion.json -F project=myproject -F version=r23 -F egg=@myproject.egg

POST
http://127.0.0.1:6800/addversion.json
{"project":"weibo","version":"2022-03-10T15_37_20","egg":file}

{
    "node_name": "***",
    "status": "ok",
    "project": "weibo",
    "version": "2022-03-22T17_37_20",
    "spiders": 11
}

八、delproject.json

删除project，支持POST提交方式

curl http://localhost:6800/delproject.json -d project=weibo

POST
http://127.0.0.1:6800/delproject.json
{"project":"weibo"}

{
    "node_name": "***",
    "status": "ok"
}

九、schedule.json

添加执行spider的job，支持POST提交方式

curl http://localhost:6800/schedule.json -d project=weibo -d spider=weibo

POST 
http://127.0.0.1:6800/schedule.json
{"setting":"a=b","project":"weibo","spider":"weibo","_version":"","priority":10}

{
    "node_name": "***",
    "status": "ok",
    "jobid": "db38e11eaa8a11ecbe813e9117ec1170"
}

十、cancel.json

取消正在执行的job，支持POST提交方式

curl http://localhost:6800/cancel.json -d project=weibo -d job=6487ec79947edab326d6db28a2d86511e8247444

POST
http://127.0.0.1:6800/cancel.json
{"project":"weibo","job":1923,"signal":"TERM"}

{
    "node_name": "***",
    "status": "ok",
    "prevstate": null
}

---

总结

scrapyd接口方便查询scrapyd服务的实时状态。