How to do concurrent async image download in PyQt5

We need to download images!

Suppose you are building a UI widget that accepts a list of image URLS. When asked, the widget is supposed to download every image, scale it, and set it as an icon for a button.

A naive approach might be to use Python’s https://pypi.org/project/requests/ package to make a get request inside of a for loop, like this:

class MyContainerClass:
   def build_ui(self, data):
       for url in urls:
          res_img = requests.get(url)
     
          # waiting, waiting, waiting...

          pixmap = QPixmap()
          pixmap.loadFromData(res_img.content)
          pixmap = pixmap.scaled(240, 140)
            
          myButton = QtWidgets.QPushButton()
          myButton.setIcon(QtGui.QIcon(pixmap))
          myButton.clicked.connect(self.select_img)

This approach exposes you to a litany of problems:

The UI will be totally blocked which the images are downloaded
The images could be large, the Internet connection spotty, and each one could take a long time
The images are downloaded one-after-the-other, meaning that your total download time will be the sum of the time it takes for every single request!

We can do better!

What is the fundamental nature of the problem?

The `requests` package downloads content synchronously (one-at-a-time), and we want to download everything in parallel.

How do I fix it?

There are many solutions to this problem. One approach is to use https://pypi.org/project/grequests/ to spawn an event loop, where each request can be fired off at the same time, downloaded in parallel, and then processed in it returns.

Since we’re using PyQt5, which is backed by the excellent Qt UI library, we already have pre-existing classes that do exactly what we want.

Let’s skill up before we proceed

Read up on these Qt classes, if you’re not already familiar with them:

QNetworkAccessManager (QNAM): https://doc.qt.io/qt-5/qnetworkaccessmanager.html
QNetworkReply: https://doc.qt.io/qt-5/qnetworkreply.html
Qt signals and slots: https://doc.qt.io/qt-5/signalsandslots.html

We will be using QNAM’s `get` method to fire off the request.

** Here’s where this tuturial is different than all the other examples I found **

Instead of listening to QNAM’s “finished” signal, which will fire repeatedly for every single image, we’re going to connect a function to QNetworkReply’s `finished` signal, which you can see in the Qt documentation:

Architecture of the solution

We’re going to create a downloader class that inherits from QObject—that way, if we set this downloader as a child of the UI widget it’s supposed to update, the downloader will be able to access that widget to set its image. As a bonus, Qt will take care of memory management, so that the downloader object will not be garbage-collected which the async downloads are running in the background.

The downloader will keep track of its own QNetworkReply, so that it can pull the image data out of the response when the `finished` signal fires.

Finally, we will instantiate our own QNAM instance and attach it to the container class. This QNAM will be passed into each downloader instance, so they can schedule their own URL download.

Let’s take a look at the solution

Image downloader class (net_io.py):

from PyQt5.QtCore import pyqtSignal, QObject
from PyQt5.QtGui import QPixmap, QIcon


class ImgDownloader(QObject):
    def __init__(self, parent, req):
        self.req = req
        super(ImgDownloader, self).__init__(parent)

    def set_button_image(self, img_binary):
        pixmap = QPixmap()
        pixmap.loadFromData(img_binary)
        pixmap = pixmap.scaled(240, 140)
        self.parent().setIcon(QIcon(pixmap))

    def start_fetch(self, net_mgr):
        self.fetch_task = net_mgr.get(self.req)
        self.fetch_task.finished.connect(self.resolve_fetch)

    def resolve_fetch(self):
        the_time = time()
        the_reply = self.fetch_task.readAll()
        self.set_button_image(the_reply)

And here’s the new main loop:

from .net_io import ImgDownloader


class myContainerClass():
    def __init__(self):
        self.download_queue = QtNetwork.QNetworkAccessManager()

    def build_ui(self, urls):
        for url in urls:
            myButton = QtWidgets.QPushButton()
            myButton.clicked.connect(self.select_img)

            req = QtNetwork.QNetworkRequest(QUrl(url))
            downloader = ImgDownloader(self.form.btn, req)
            downloader.start_fetch(self.download_queue)

What did we just accomplish?

The QNAM download utilizes Qt signals and slots, so it does the actual download work “off the main thread”—which means that our UI is still responsive.
The downloads are accomplished in parallel, so we are no longer blocked by having to download them one-at-a-time, especially if some of the images download more slowly.

All of the download requests are fired off immediately. QNAM rate limits these to 6 connections at a time, and this can be easily configured if desired.
The QNAM is managed by the container class, and each ImgDownloader instance is managed by the UI element (myButton) that it pertains to.

Suggestions or Comments?

Tweet at https://twitter.com/raphaeltraviss so that I can get this blog post updated—it takes a village to raise a developer.