Skip to main content

Set Up Backend

To be able to build and work on the backend components, it is necessary to have Python and Docker Compose installed.

Python​

Since most of the backend is written in Python, it is necessary to have a Python interpreter installed. Please refer here for the latest releases. It is recommended to have at least Python 3.11 installed for this project.

Also, it is recommended to have a virtual environment set up for the project. One possibility for that is installing conda. Another option is to just create a venv. Using a virtual environment has the advantage to provide an own environment where you can install all necessary Python packages without having those in your general Python environment.

Docker / Docker Compose​

The whole project is executed via Docker containers. Because of this, it is necessary to have Docker and Docker Compose installed on your machine when working on it.

Docker separates the different components of the project into so-called containers. Each of those containers acts on an own virtualization layer. With Docker Compose, all those specific containers can be built and act together as one project.

A guide on how to install Docker on Linux can be found here. On Windows, Docker Desktop can be installed. Please keep in mind that on Windows WSL2 also needs to be installed for Docker to work.

After Docker is installed, Docker Compose can be installed using this guide.

Developer Mode​

The developer mode is a setting that can be enabled to limit the number of firmware images that are downloaded per vendor. This is useful for development and testing, as it reduces the time and disk space required to run the scraper.

To enable the developer mode, pass --developer_mode=y to the configure script, or answer y when prompted during an interactive run:

# Non-interactive
./configure --developer_mode=y

# Combined with headless mode
./configure --headless --developer_mode=y

This will set the developer_mode variable in the .env and config.json files to True.

When the developer mode is active, the scraper will only download 3 firmware images per vendor. The limit is defined in the firmware_scraper/settings.py file.