Skip to main content

Testing Pipeline

This document describes the architecture of the automated testing environment, the logic behind the integration tests, and the configuration of the CI/CD pipeline (/.gitlab-ci.yml).

Pipeline Architecture & GitLab CI​

The pipeline was upgraded to support Full-Stack Integration Testing. Unlike unit tests, these integration tests verify the interaction between the Python logic, the MySQL database, and the Redis message broker.

Docker registry and image management
To ensure a consistent and fast environment, a custom Docker image is build:

  • Dockerfile: Located at /tests/integration/Dockerfile.
  • Base: python:3.11-slim.
  • Version Pinning: apt and python packages are pinned to specific versions. See /tests/integration/all_requirements.txt
  • Registry Upload: The image is built by the pipeline jobs generate-requirements and build_base_test and pushed to the GitLab Container Registry at hub.informatik.hs-augsburg.de.

Environment Variables & PYTHONPATH
A critical configuration for the imports to work across the project structure is the PYTHONPATH.

  • Variable: PYTHONPATH: "$CI_PROJECT_DIR/jobs"
  • Purpose: By setting this in the CI variables or the Dockerfile, Python can resolve modules like from analysis_tool_set.redis_communication import RedisTaskTracker or from DBConnector import DBConnector correctly, even when the test runner is executed from the root directory.

Integration Test Logic​

The integration tests are located in /tests/integration/. They use a Live-Service approach, meaning they talk to real mysql:8 and redis:7-alpine containers started as GitLab services.

Modules Tested
Basically modules from /jobs are tested.

  • /jobs/analysis_tool_set/redis_communication.py: The handler for the redis queues.
  • /jobs/DBConnector.py: The database abstraction layer.
  • /jobs/executor/executor.py: The core Flask API and the "Splitter" logic.
  • tools which are implemented from the Template_Analysis_Tool. Currently tested are /jobs/extractor/extractor.py, /jobs/cve_bin_tool/cve_bin_tool.py, /jobs/cve_bin_tool/sbom_gen.py, /jobs/firmwalker/firmwalker.py and //jobs/trufflehog/trufflehog.py.
  • /jobs/scheduler.py: Manages the new jobs into queueing.
  • /jobs/api.py: The api which connects the frontend with the executor and scheduler.

Test Dependencies
The order of the tests was created in order to only test modules if all other modules they depend on were already tested. The dependencies can be reconstructed by simply following the modules imports.
The dependencies for all listed modules are as follows.

  • database > none
  • redis > none
  • tools > database, redis
  • executor > redis, tools
  • scheduler > executor
  • api > scheduler

Summary: - (database, redis) -> tools -> executor -> - scheduler -> api
While some endpoints are mocked to test the modules in isolation, tests will (try to) only use the functionality of other modules if they were previously tested.

Database Preparation​

Before tests run, the database is initialized using /docker_compose_dir/sql_scripts/init.sql.

  • Tables: Creates products, jobs, tasks, tools, tool_dependencies, etc.
  • Seeding: In the .db-setup stage of the pipeline, the mysql client is used to pipe SQL into the container.
  • Wait-Loop: The pipeline includes a timeouted loop which waites for the database to start. This prevents the tests from starting before the MySQL service is fully initialized and ready to accept connections.

Writing basic tests for a new tool​

The file test_tools.py under /tests/integration needs to be updated if a new tool is added to the code base to guarantee coverage of the test functions. All tools have (currently) two basic test cases.

  • The first tests reimports the tool and checks the database "tools" for whether the tool has successfully verified with the database (testing the @tool(...) decorator). If the tool has any dependencies, the tool_dependencies should also be verified.
  • The second test checks how the tool will handle a task. It manually creates a payload, and test whether the run_tool function throws any errors.

Relevant test's should also be updated, whenever functionality changes (e.g. when jobs were enabled to handle multiple tools, i.e. tool -> tools).

warning

Testing functions must be named with the prefix test_ otherwise, the pytest command will not recognize these as functions to be executed during testing!

build job for the python base​

The "prepare" stage is generating a python image which searched for all requirements in the project (all requirements.txt files), uploads it to the GitLab registry and uses this base for all tests. This ensures all tests can use all defined libraries.