开源代码学习之persepolis【二】

 

一、下载界面

 

1、主界面下载显示

MainWindow首先对aria进行初始化，启动aria2。启动方法在download.py中。

 # start aria2
        start_aria = StartAria2Thread()
        self.threadPool.append(start_aria)
        self.threadPool[0].start()
        self.threadPool[0].ARIA2RESPONDSIGNAL.connect(self.startAriaMessage)

然后定义添加下载链接界面，AddLinkWindow可以有多个，都存放在self.addlinkwindows_list中。

调用的时候有三个参数（self, self.callBack, self.persepolis_setting），下载添加界面通过回调函数传递参数给主界面的callBack函数。callBack获取下载信息后，添加到线程池中。

new_download = DownloadLink(gid, self)
            self.threadPool.append(new_download)
            self.threadPool[len(self.threadPool) - 1].start()
            self.threadPool[len(self.threadPool) -
                            1].ARIA2NOTRESPOND.connect(self.aria2NotRespond)

注意主界面的addLinkSpiderCallBack函数，该函数调用顺序为：

1、下载添加界面获取下载链接改变（linkLineChanged）信息

2、下载添加界面开启线程AddLinkSpiderThread尝试获取链接文件大小，通过parent将该线程添加到主界面线程池中。并将AddLinkSpiderThread的信号连接到主线程的addLinkSpiderCallBack函数，同时将下载添加界面的指针child添加到槽函数的参数中，这样主界面可以通过child访问下载添加界面。

self.parent.threadPool[len(self.parent.threadPool) - 1].ADDLINKSPIDERSIGNAL.connect(
                partial(self.parent.addLinkSpiderCallBack, child=self))

3、AddLinkSpiderThread线程将结果ADDLINKSPIDERSIGNAL信号发送给主界面addLinkSpiderCallBack函数，注意这里发射的时候，只有dict参数，连接的时候有两个参数。

self.ADDLINKSPIDERSIGNAL.emit(spider_dict)

4、主界面addLinkSpiderCallBack函数通过child调用下载添加界面，设置文件名称和大小的显示。

这样就是下载链接界面新增线程到主界面，然后主界面线程执行完成后控制子界面更新，为什么不是下载链接添加界面自己开启一个线程获取文件大小，然然后根据获取结果自己改变下载链接界面呢？

 

mainwindow.py:

class DownloadLink(QThread):
    ARIA2NOTRESPOND = pyqtSignal()

    def __init__(self, gid, parent):
        QThread.__init__(self)
        self.gid = gid
        self.parent = parent

    def run(self):
        # add gid of download to the active gids in temp_db
        # or update data base , if it was existed before
        try:
            self.parent.temp_db.insertInSingleTable(self.gid)
        except:
            # release lock
            self.parent.temp_db.lock = False
            dictionary = {'gid': self.gid, 'status': 'active'}
            self.parent.temp_db.updateSingleTable(dictionary)

        # if request is not successful then persepolis is checking rpc
        # connection with download.aria2Version() function
        answer = download.downloadAria(self.gid, self.parent)
        if answer == False:
            version_answer = download.aria2Version()

            if version_answer == 'did not respond':
                self.ARIA2NOTRESPOND.emit()


class MainWindow(MainWindow_Ui):
    def __init__(self, start_in_tray, persepolis_main, persepolis_setting):
        super().__init__(persepolis_setting)
        self.persepolis_setting = persepolis_setting
        self.persepolis_main = persepolis_main


        # list of threads
        self.threadPool = []

        # start aria2
        start_aria = StartAria2Thread()
        self.threadPool.append(start_aria)
        self.threadPool[0].start()
        self.threadPool[0].ARIA2RESPONDSIGNAL.connect(self.startAriaMessage)


    def addLinkButtonPressed(self, button=None):
         addlinkwindow = AddLinkWindow(self, self.callBack, self.persepolis_setting)
         self.addlinkwindows_list.append(addlinkwindow)
         self.addlinkwindows_list[len(self.addlinkwindows_list) - 1].show()

    # callback of AddLinkWindow
    def callBack(self, add_link_dictionary, download_later, category):
        # write information in data_base
        self.persepolis_db.insertInDownloadTable([dict])
        self.persepolis_db.insertInAddLinkTable([add_link_dictionary])

        # if user didn't press download_later_pushButton in add_link window
        # then create new qthread for new download!
        if not(download_later):
            new_download = DownloadLink(gid, self)
            self.threadPool.append(new_download)
            self.threadPool[len(self.threadPool) - 1].start()
            self.threadPool[len(self.threadPool) -
                            1].ARIA2NOTRESPOND.connect(self.aria2NotRespond)

            # open progress window for download.
            self.progressBarOpen(gid)

            # notify user
            # check that download scheduled or not
            if not(add_link_dictionary['start_time']):
                message = QCoreApplication.translate("mainwindow_src_ui_tr", "Download Starts")
            else:
                new_spider = SpiderThread(add_link_dictionary, self)
                self.threadPool.append(new_spider)
                self.threadPool[len(self.threadPool) - 1].start()
                self.threadPool[len(self.threadPool) - 1].SPIDERSIGNAL.connect(self.spiderUpdate)
                message = QCoreApplication.translate("mainwindow_src_ui_tr", "Download Scheduled")
            notifySend(message, '', 10000, 'no', parent=self)


    # see addlink.py file
    def addLinkSpiderCallBack(self, spider_dict, child):
        # get file_name and file_size
        file_name = spider_dict['file_name']
        file_size = spider_dict['file_size']

        if file_size:
            file_size = 'Size: ' + str(file_size)
            child.size_label.setText(file_size)

        if file_name and not(child.change_name_checkBox.isChecked()):
            child.change_name_lineEdit.setText(file_name)
            child.change_name_checkBox.setChecked(True)

 

2、下载添加界面

下载添加界面AddLinkWindow将第一个参数self初始化为parent，后续通过该参数对主界面进行访问，第二个参数为回调函数，用于传递参数给主界面，第三个参数将系统设置传递给下载添加界面。

在下载链接改变时，将AddLinkSpiderThread加入到主界面的threadPool中，并将ADDLINKSPIDERSIGNAL连接到主界面的addLinkSpiderCallBack。

new_spider = AddLinkSpiderThread(dict)

self.parent.threadPool.append(new_spider)

self.parent.threadPool[len(self.parent.threadPool) - 1].ADDLINKSPIDERSIGNAL.connect(
                partial(self.parent.addLinkSpiderCallBack, child=self))

AddLinkSpiderThread通过spider.addLinkSpider获取到文件大小和名称信息，发送给主界面的addLinkSpiderCallBack函数，注意这里发射的时候，只有dict参数，连接的时候有两个参数。

self.ADDLINKSPIDERSIGNAL.emit(spider_dict)

在按下确定按钮后，通过callback回调函数调用传递参数给主界面。

addlink.py:

class AddLinkSpiderThread(QThread):
    ADDLINKSPIDERSIGNAL = pyqtSignal(dict)

    def __init__(self, add_link_dictionary):
        QThread.__init__(self)
        self.add_link_dictionary = add_link_dictionary

    def run(self):
        try:
            # get file name and file size
            file_name, file_size = spider.addLinkSpider(self.add_link_dictionary)

            spider_dict = {'file_size': file_size, 'file_name': file_name}

            # emit results
            self.ADDLINKSPIDERSIGNAL.emit(spider_dict)

class AddLinkWindow(AddLinkWindow_Ui):
    def __init__(self, parent, callback, persepolis_setting, plugin_add_link_dictionary={}):
        super().__init__(persepolis_setting)
        self.callback = callback
        self.plugin_add_link_dictionary = plugin_add_link_dictionary
        self.persepolis_setting = persepolis_setting
        self.parent = parent    

        self.link_lineEdit.textChanged.connect(self.linkLineChanged)

        self.ok_pushButton.clicked.connect(partial(
             self.okButtonPressed, download_later=False))
        self.download_later_pushButton.clicked.connect(
             partial(self.okButtonPressed, download_later=True))

    # enable when link_lineEdit is not empty and find size of file.
    def linkLineChanged(self, lineEdit):
        if str(self.link_lineEdit.text()) == '':
            self.ok_pushButton.setEnabled(False)
            self.download_later_pushButton.setEnabled(False)
        else:  # find file size

            dict = {'link': str(self.link_lineEdit.text())}

            # spider is finding file size
            new_spider = AddLinkSpiderThread(dict)
            self.parent.threadPool.append(new_spider)
            self.parent.threadPool[len(self.parent.threadPool) - 1].start()
            self.parent.threadPool[len(self.parent.threadPool) - 1].ADDLINKSPIDERSIGNAL.connect(
                partial(self.parent.addLinkSpiderCallBack, child=self))

            self.ok_pushButton.setEnabled(True)
            self.download_later_pushButton.setEnabled(True)
    def okButtonPressed(self, button, download_later):
        # user submitted information by pressing ok_pushButton, so get information
        # from AddLinkWindow and return them to the mainwindow with callback!


        # save information in a dictionary(add_link_dictionary).
        self.add_link_dictionary = {'referer': referer, 'header': header, 'user_agent': user_agent, 'load_cookies': load_cookies,
                                    'out': out, 'start_time': start_time, 'end_time': end_time, 'link': link, 'ip': ip,
                                    'port': port, 'proxy_user': proxy_user, 'proxy_passwd': proxy_passwd,
                                    'download_user': download_user, 'download_passwd': download_passwd,
                                    'connections': connections, 'limit_value': limit, 'download_path': download_path}

        # get category of download
        category = str(self.add_queue_comboBox.currentText())

        del self.plugin_add_link_dictionary

        # return information to mainwindow
        self.callback(self.add_link_dictionary, download_later, category)

        # close window
        self.close()

3、总结

1、线程间传递参数可以通过回调函数传递，也可以通过信号和槽传递。

2、主从线程之间，主线程将self传递给从线程，从线程可以对主线程的函数进行调用。从线程也可以将self传递给主线程，由主线程对从线程进行函数调用

 

二、下载文件

 

启动aria2的服务是通过subprocess.Popen启动的，每个选项的意义在aria2接口文档都有介绍。

subprocess 模块允许你生成新的进程，连接它们的输入、输出、错误管道，并且获取它们的返回码。此模块打算代替一些老旧的模块与功能os.system, os.popen*, os.spawn.

https://docs.python.org/zh-cn/3/library/subprocess.html#subprocess.Popen.communicate

https://blog.csdn.net/qq_34355232/article/details/87709418

subprocess.Popen([aria2d, '--no-conf', '--enable-rpc', '--rpc-listen-port=' + str(port), '--rpc-max-request-size=2M', '--rpc-listen-all', '--quiet=true'], stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE, shell=False, creationflags=NO_WINDOW)

 

添加下载链接是通过XML-RPC远程调用完成的：

server = xmlrpc.client.ServerProxy(server_uri, allow_none=True)

aria2的RPC接口介绍如下，支持JSON-RPC和XML-RPC。

https://aria2.github.io/manual/en/html/aria2c.html#rpc-interface

python的XML-RPC库介绍文档很多，找了两个如下：

https://www.jianshu.com/p/9987913cf734

https://developer.51cto.com/art/201906/597963.htm

GID：aria2通过GID索引管理每个下载，GID为64位二进制数。RPC访问时，表示为长度16个字符的十六进制字符串。通常aria2为每个下载链接产生衣蛾GID，用户也可以通过GID选项指定。

通过XML-RPC访问aria2

aria2.addUri([secret, ]uris[, options[, position]])¶

添加下载的链接，URIS是下载链接数组，option，positon是一个整数，表示插在下载队列的位置，0表示第一个。如果没有提供position参数或者position比队列的长度长，则添加的下载在下载队列的最后。该方法返回新注册下载的GID。

aria2.tellStatus([secret, ]gid[, keys])

该方法返回指定下载GID的进展，keys是一个字符串数组，指定了需要查询哪些项目。如果keys为空或者省略，则包含所有的项目。常用的项目有gid、status、totalLength、completedLength、downloadSpeed、uploadSpeed、numSeeders、connections、dir、files。

aria2.tellActive([secret][, keys])

该方法查询激活下载的状态，查询的项目与aria2.tellStatus类似。

aria2.removeDownloadResult([secret, ]gid)

根据GID从存储中移除下载完成/下载错误/删除的下载，如果成功返回OK

aria2.remove([secret, ]gid)

根据GID删除下载，如果下载正在进行先停止该下载。该下载链接的状态变为removed状态。返回删除状态的GID。

aria2.pause([secret, ]gid)

暂停指定GID的下载链接，下载链接的状态变为paused。如果下载是激活的，则该下载链接放置在等待队列的最前面。要想将状态变为waiting，需要用aria2.unpause方法。

download.py

# get port from persepolis_setting
port = int(persepolis_setting.value('settings/rpc-port'))

# get aria2_path
aria2_path = persepolis_setting.value('settings/aria2_path')

# xml rpc
SERVER_URI_FORMAT = 'http://{}:{:d}/rpc'
server_uri = SERVER_URI_FORMAT.format(host, port)
server = xmlrpc.client.ServerProxy(server_uri, allow_none=True)

# start aria2 with RPC


def startAria():
    
    # in Windows
    elif os_type == OS.WINDOWS:
        if aria2_path == "" or aria2_path == None or os.path.isfile(str(aria2_path)) == False:
            cwd = sys.argv[0]
            current_directory = os.path.dirname(cwd)

            aria2d = os.path.join(current_directory, "aria2c.exe")  # aria2c.exe path
        else:
            aria2d = aria2_path

        # NO_WINDOW option avoids opening additional CMD window in MS Windows.
        NO_WINDOW = 0x08000000

        if not os.path.exists(aria2d):
            logger.sendToLog("Aria2 does not exist in the current path!", "ERROR")
            return None
        # aria2 command in windows
        subprocess.Popen([aria2d, '--no-conf', '--enable-rpc', '--rpc-listen-port=' + str(port),
                          '--rpc-max-request-size=2M', '--rpc-listen-all', '--quiet=true'],
                         stderr=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stdin=subprocess.PIPE,
                         shell=False,
                         creationflags=NO_WINDOW)

    time.sleep(2)

    # check that starting is successful or not!
    answer = aria2Version()

    # return result
    return answer

# check aria2 release version . Persepolis uses this function to
# check that aria2 RPC connection is available or not.


def aria2Version():
    try:
        answer = server.aria2.getVersion()
    except:
        # write ERROR messages in terminal and log
        logger.sendToLog("Aria2 didn't respond!", "ERROR")
        answer = "did not respond"

    return answer

def downloadAria(gid, parent):
    # add_link_dictionary is a dictionary that contains user download request
    # information.

    # get information from data_base
    add_link_dictionary = parent.persepolis_db.searchGidInAddLinkTable(gid)

    answer = server.aria2.addUri([link], aria_dict)

三、数据库

使用sqlite3数据库教程：https://docs.python.org/zh-cn/3/library/sqlite3.html

有三个数据库TempDB在内存中，放置实时数据，PluginsDB放置浏览器插件传来的新链接数据，PersepolisDB是主要的数据库，存放下载信息。

TempDB有两个表，single_db_table存放下载中的GID，queue_db_table存放下载队列的GID信息。

PersepolisDB有四个表：

category_db_table存放类型信息，包括'All Downloads'、'Single Downloads'和'Scheduled Downloads'类型。

download_db_table存放主界面显示的下载状态表。

addlink_db_table存放下载添加界面添加的下载链接。

video_finder_db_table存放下载添加界面添加下载的信息。

# This class manages TempDB
# TempDB contains gid of active downloads in every session.
class TempDB():
    def __init__(self):
        # temp_db saves in RAM
        # temp_db_connection

        self.temp_db_connection = sqlite3.connect(':memory:', check_same_thread=False)

   def createTables(self):
        # lock data base
        self.lockCursor()
        self.temp_db_cursor.execute("""CREATE TABLE IF NOT EXISTS single_db_table(

        self.temp_db_cursor.execute("""CREATE TABLE IF NOT EXISTS queue_db_table(


# persepolis main data base contains downloads information
# This class is managing persepolis.db
class PersepolisDB():
    def __init__(self):
        # persepolis.db file path
        persepolis_db_path = os.path.join(config_folder, 'persepolis.db')

        # persepolis_db_connection
        self.persepolis_db_connection = sqlite3.connect(persepolis_db_path, check_same_thread=False)

    # queues_list contains name of categories and category settings
    def createTables(self):

        # lock data base
        self.lockCursor()
        # Create category_db_table and add 'All Downloads' and 'Single Downloads' to it
        self.persepolis_db_cursor.execute("""CREATE TABLE IF NOT EXISTS category_db_table(

        # download table contains download table download items information
        self.persepolis_db_cursor.execute("""CREATE TABLE IF NOT EXISTS download_db_table(

        # addlink_db_table contains addlink window download information
        self.persepolis_db_cursor.execute("""CREATE TABLE IF NOT EXISTS addlink_db_table(


        # video_finder_db_table contains addlink window download information
        self.persepolis_db_cursor.execute("""CREATE TABLE IF NOT EXISTS video_finder_db_table(

sqlite3 模块支持两种占位符：问号（qmark风格）和命名占位符（命名风格）。

# This is the qmark style:

cur.execute("insert into people values (?, ?)", (who, age))

# And this is the named style:

cur.execute("select * from people where name_last=:who and age=:age", {"who": who, "age": age})

coalesce函数返回其参数中第一个非空表达式的值，也即提供了参数则用新参数，未提供新参数则用原值。

self.temp_db_cursor.execute("""UPDATE single_db_table SET shutdown = coalesce(:shutdown, shutdown),
                                                                status = coalesce(:status, status)
                                                                WHERE gid = :gid""", dict)

MainWindow在初始化时创建CheckDownloadInfoThread线程，轮询每一个下载中的链接，并将结果返回给主界面的checkDownloadInfo函数进行下载状态更新。

        # CheckDownloadInfoThread
        check_download_info = CheckDownloadInfoThread(self)
        self.threadPool.append(check_download_info)
        self.threadPool[1].start()
        self.threadPool[1].DOWNLOAD_INFO_SIGNAL.connect(self.checkDownloadInfo)
        self.threadPool[1].RECONNECTARIASIGNAL.connect(self.reconnectAria)