Overview
This project aims to automate firmware analysis as much as possible. Firmware_Vault independently collects and catalogs firmware images from device manufacturer websites using the Python framework Scrapy.
With a user-friendly interface, a great variety of analysis jobs can be created and executed on the firmware images to scan for vulnerabilities. Users have great flexibility in choosing analysis methods (e.g. binwalk, cve-bin-tool, etc.) and the ability to run jobs repeatedly as cronjobs. This way, a wide collection of various firmware can be quickly scanned for vulnerabilities.
Scraping
The scraping process is started by the scheduler container, which starts the Python scraper container. The scraper then pulls the latest firmware from the enabled vendors. Metadata related to the firmware, like the version, release date and product name, is inserted in the database, while the firmware is saved to the local file system. It also attempts to extract the .bin file from the downloaded zip files.
Frontend
To simplify user interactions, a web user interface is provided, which can be accessed by visiting port 80 (HTTP) of the host system Firmware_Vault is running on. There, the user can interact with the three main parts of the UI:
Analysis Engine
The Analysis Engine performs security testing of the firmware catalog. The Analysis Engine mainly consists of three parts running in two separate containers.
api_scheduler Container
This container hosts the API as well as the scheduler.
api.py:
The api.py acts as the backend and works on requests from the web UI. Requests can be data requests from the DB, running SQL queries, or the creation of a new job.
scheduler.py:
The scheduler decides when a job is handed to the executor. To avoid executing a job multiple times the scheduler checks if the job is already running, before handing a job to the executor. All jobs are executed at once immediately, unless a cron schedule was provided, which holds the job in the api_scheduler until the job is executed according to the schedule.
Consult the analysis engine documentation page for additional information.
Executor Container
The analysis tools, like binwalk, will be executed in this container. To use new analysis tools, they will have to be loaded onto the executor container using the executor's Dockerfile.
The executor.py program runs in this container. The executor receives only signals, so it pulls job data from the database. The execution of the job tasks is done in parallel and the output of the analysis tool is placed into an output.txt in the file system (volume: executor).
Docker Containers
All components of Firmware_Vault run in Docker containers. To avoid lengthy setup, the containers are combined in a Docker Compose file.
Data that needs to be stored in a file system should be stored in a Docker volume that references a folder in the /docker_compose_dir/volumes.
Additional information can be accessed at the projects' Docker documentation page or the official Docker documentation page.