Skip to main content

Docker

For a basic introduction to docker please refer to the official documentation

Structure​

The directory docker_compose_dir contains the docker-compose.yaml that manages the project. Each Unit of the project is contained within a docker container.

Containers:

  • mysql_db
    Central data storage of the project
  • backup
    Periodically creates a backup of the mysql container
  • python_scraper
    Fetches firmware images from vendor websites
  • scheduler
    Periodically run scraper
  • frontend
    Website for the Firmware_vault project
  • api_scheduler
    Handles job scheduling for the firmware analysis
  • executor
    Handles the execution of scheduled jobs
  • redis
    Stores the job queue
  • binwalk (github Repository)
    (Analysis Tool) Unpacks the downloaded firmware images
  • cve-bin-tool (pip Package)
    (Analysis Tool) Finds known vulnerabilities in software
  • firmwalker
    (Analysis Tool) Helps find things of interest, e.g. passwords, in mounted firmware images

Develop with docker watch​

The file docker_compose_dir/compose.dev.yaml contains watch statements that enable quicker development.

To include the watch statements use this command:

docker compose -f docker-compose.yaml -f compose.dev.yaml up --watch

Upon changes in files that are included in the compose.dev.yaml, the files get synced into the container. There are three available settings:

  • sync
    The file gets synced into the container,
  • sync+restart
    The file gets synced into the container. The container is then restarted,
  • rebuild
    Upon changes on the file, the container will be rebuilt.

This removes the need for manual container restarts during development.

Debugging Docker Problems​

(Please note the Docker compose can take up to 20 minutes to build fresh)

While Docker has a bunch of advantages, it can be incredibly frustrating to debug. Here are some issues we encountered and how we fixed them.

Volumes are unable to find their folders​

After calling "docker compose up," you might get an error like this:

Error response from daemon: 
error while mounting volume 'volume path in container/_data': 
failed to mount local volume: mount *path to the project*/docker_compose_dir/volumes/*one 
of the volumes*: volume path in container/_data, flags: 0x1000: no such file or directory

This happens if you move the project directory to a different location after starting the docker environment at least once. Docker saves the volumes with absolute paths and does not update these if you move the project folder, therefore you need to delete the volumes and recreate them.

How to fix:

This command will delete all volumes from Docker so they can be rebuilt: !! If you have volumes from other projects, they will also be deleted !!

docker volume rm $(docker volume ls -q)

You can find the names of all volumes in the docker_compose_dir/docker-compose.yaml. To remove a specific volume, e.g. backup use this command:

docker volume rm docker_compose_dir_backup

This will only work if the volumes are not used by a container. If the volume is still in use, you have to delete the container currently using the volume

My SQL DB container is in a restarting loop​

This is probably caused because the mysql_data volume is not empty (and the data in it conflicts with the init.sql).

How to fix: Delete the contents of the /docker_compose_dir/volumes/mysql_data folder (hint: MySQL also creates hidden files).

If this does not work, also try to delete the volumes from Docker (so it can be rebuilt cleanly):

docker volume rm mysql_data

Clear everything from Docker​

Stop all running containers​

docker stop $(docker ps -aq)

Remove all containers​

docker rm $(docker ps -aq)

Remove all images​

docker rmi $(docker images -q)

Remove all volumes​

docker volume rm $(docker volume ls -q)

Remove all networks (excluding the default ones)​

docker network rm $(docker network ls | grep -v "bridge\|host\|none" | awk '/ / { print $1 }')

Remove Docker build cache​

docker builder prune -a