使用Python在网页上下载所有链接（相关文档）

Question

我必须从网页上下载很多文档。它们是wmv文件，PDF，BMP等。当然，它们都具有指向它们的链接。因此，每次我必须RMC一个文件，选择“链接另存为”，然后保存，然后键入“所有文件”。是否可以在Python中执行此操作？我搜索了SO
DB，人们回答了如何从网页获取链接的问题。我想下载实际文件。提前致谢。（这不是硬件问题：））。

Answer 1

这是如何从http://pypi.python.org/pypi/xlwt下载一些选定文件的示例

您将需要先安装机械化：http :
//wwwsearch.sourceforge.net/mechanize/download.html

    import mechanize
    from time import sleep
    #Make a Browser (think of this as chrome or firefox etc)
    br = mechanize.Browser()

    #visit http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
    #for more ways to set up your br browser object e.g. so it look like mozilla
    #and if you need to fill out forms with passwords.

    # Open your site
    br.open('http://pypi.python.org/pypi/xlwt')

    f=open("source.html","w")
    f.write(br.response().read()) #can be helpful for debugging maybe

    filetypes=[".zip",".exe",".tar.gz"] #you will need to do some kind of pattern matching on your files
    myfiles=[]
    for l in br.links(): #you can also iterate through br.forms() to print forms on the page!
        for t in filetypes:
            if t in str(l): #check if this link has the file extension we want (you may choose to use reg expressions or something)
                myfiles.append(l)


    def downloadlink(l):
        f=open(l.text,"w") #perhaps you should open in a better way & ensure that file doesn't already exist.
        br.click_link(l)
        f.write(br.response().read())
        print l.text," has been downloaded"
        #br.back()

    for l in myfiles:
        sleep(1) #throttle so you dont hammer the site
        downloadlink(l)

注意：在某些情况下，您可能希望替换br.click_link(l)为br.follow_link(l)。区别在于click_link返回一个Request对象，而follow_link将直接打开该链接。