proxmox-test-tools

proxmox-test-tools is the engine of Proxmox's integration-test system: it provisions throwaway test instances, runs test cases against them, and coordinates CI runs. The test cases themselves live in the sibling repository proxmox-e2e-tests (the content); the only coupling between the two is a JSON file the runner injects into each test via $CONFIG_PATH (see Test case declarations).

Crates

Crate	Role
proxmox-test-common	Shared types (TestInstance, Virtualization, TestInventory, Secret), RON loading, secret resolution, CLI logging, and the run_with_timeout command helper.
proxmox-test-instance	Provisions and manages test VMs and containers on a PVE host (clone or unattended ISO install), forms clusters, snapshots, rolls back, and registers PDM remotes.
proxmox-test-runner	Runs the test cases from a testcase file against the instances described by an inventory file.
proxmox-test-scheduler	HTTPS daemon that coordinates CI runs over SSH: sets up instances, installs the packages under test, runs the runner, and mails the report.

Pipeline

Plan (*.ron)           --proxmox-test-instance setup-->  Inventory (*.ron)
                                                              |
TestcaseConfig (*.ron) --proxmox-test-runner run-->  per-test JSON ($CONFIG_PATH)
                                                              |
                                                              v
                                             report (text / JSON / JUnit / Markdown)

proxmox-test-instance

TL;DR

• Helper tool that can effortlessly manage test instances
  • Either by cloning some existing instance template
  • Or by using the auto-installer to automatically install a PVE/PBS/PMG test instance
• Creates an inventory file that can be later fed into proxmox-test-runner

Setup

• A (preferably) physical PVE installation which hosts the test runner

• A config file at ~/.config/proxmox-test-tools/proxmox-test-instance.ron, see example/proxmox-test-instance.ron for reference.

• (most likely) ~/.config/proxmox-test-tools/secrets.ron for storing secrets, unless all setup plan/inventory files use inline secrets - see example/secrets.ron

• An access token needs to be specified in the config file (for now), it needs the following roles:

  • /sdn: PVESDNAdmin (for creating the Zone, Vnet, Subnet and applying the SDN changes)
  • /storage/: PVEDatastoreUser (for the storage storing the ISOs as well as where the VM disk is allocated)
  • /vms: PVEVMAdmin (for creating/starting/stopping/deleting VMs, snapshots, etc.) The token can be stored in plain text in proxmox-test-instance.ron or provided via a command (See test_host_token_cmd in the example).

• For automatic installations (can be achieved automatically with Ansible):

  • A SDN simple zone with automatic DHCP
  • A VNet and Subnet within that zone (e.g. 172.16.0.0/24). The subnet needs a DHCP range assigned (e.g 172.16.0.100 to 172.16.0.250)
  • If the hosts has a firewall enabled, the following rules might be needed:
    • in: ACCEPT, interface , protocol: udp, dest port: 67 (DHCP)
    • in: ACCEPT, interface , macro: DNS
    • in: ACCEPT, interface , protocol: tcp, dest port: 8132 (responder for answer file)
  • ISO files configured to fetch the answer file via the subnet's gateway

proxmox-test-instance prepare-iso proxmox-ve_8.2-1.iso
cp proxmox-ve_8.2-1-auto-from-http-url.iso /var/lib/vz/template/iso/

This is equivalent to the following call to proxmox-auto-install-assistant:

proxmox-auto-install-assistant prepare-iso --fetch-from http --url http://172.16.0.1:8132/answerfile proxmox-ve_8.2-1.iso

Ansible

The Ansible playbooks in the ansible/ directory automate this setup. See ansible/README.md for details; site.yml deploys a scheduler together with its test hosts and local.yml configures a single development host.

Example Usage

To prepare all test assets from an inventory setup plan file use the proxmox-test-instance setup command. Depending on the plan, this will

• Create and start a new VM
• Start the PVE/PBS autoinstaller from a provided ISO
• Wait for the installation to finish
• Run any post-setup scripts/playbooks
• Form configured clusters and shut down the guests
• Create an offline snapshot called post-setup
• Write the current inventory state to the given inventory file, which can be used by proxmox-test-runner

# Set up test instances as defined in pve_tests/setup_basic.ron
proxmox-test-instance setup pve_tests/setup_basic.ron inventory.ron --secrets secrets.ron

Setup leaves guests stopped, and its offline snapshots restore in that state. For manual work, use proxmox-test-instance start to start the required entries, including every member before calling create-cluster.

Registering PDM remotes requires capacity for at least two guests in max_concurrent_setups. PDM stays online while remotes are registered one at a time.

After receiving a task ID, the instance tool waits up to one hour for that task, including the time spent on status requests. This is a CI deadline, not proof that a slower task is stuck, and it does not cancel the server-side task. Check its status on the host before retrying an operation that may still be running.

To run test cases, use the proxmox-test-runner run command. As an input, it takes a file containing test case definitions and reads the passed inventory file to gather information about any provided instances.

# Start each test case's guests before running it, as setup leaves them stopped
proxmox-test-runner run inventory.ron testcases.ron --start-instances true

Alternatively, you can run tests directly on the localhost without an inventory file using the run-local command:

# Run local tests (optionally filtering by tags and generating a report)
proxmox-test-runner run-local testcases.ron --tags mytag --report report.json --report-format json

After test execution, test assets can be cleaned up. This will

• force stop test asset VMs
• remove all test asset VMs
• remove the inventory file, as nothing it describes is left

# Clean up test assets
proxmox-test-instance cleanup inventory.ron

Also, the proxmox-test-instance purge command can be used to clean up the ENTIRE VMID range that was configured for the tool. As a safety-mechanism, only VMs with the proxmox-test-runner tag will be deleted.

Setup inventory declarations

Instances that should be created are configured via an instance setup plan file.

#![enable(implicit_some)]
Plan(
    instances: [
        SetupInstance(
            name: "pve1",
            kind: Vm(
                fqdn: "pve1.example.com",
                iso: "local:iso/proxmox-ve_8.2-1-auto-from-http-url.iso",
                cores: 2,
                memory: 3000,
                disk_setup: Zfs(
                    size: 10,
                    raid: Raid1,
                ),
                root_password: Ref("default"),
                // or using an inline secret:
                // root_password: Inline("password"),
            ),
            post_setup: [
                "scripts/update-packages.sh pve staging",
            ],
        ),
    ]
)

Scripts

The proxmox-test-instance run-script command can be used to run scripts on test instances, e.g. to update them to the latest packages, to inject test artifacts etc.

Setup hooks for assets are also provided with the asset configuration via a JSON configuration file, located at $CONFIG_PATH. This allows to use the same hook script for multiple assets.

{
  "host": "172.16.0.100",
  "ssh_private_keyfile": "/home/user/.ssh/id_rsa",
  "ssh_user": "root",
  "token_userid": "",
  "token_value": ""
}

To get the contents in a hook script, jq may be used. Scripts are executed on the same host where proxmox-test-instance is executed. SSH may be used to perform any setup steps on the instance itself. An example can be seen in hooks/update-packages.sh in the sibling proxmox-e2e-tests repository.

proxmox-test-runner

TL;DR

• Runs test scripts, as defined in RON configuration files against test instances provided by an inventory file
• Each test case can specify required test instances (e.g. PVE installations that are to be tested, auxiliary test hosts required in the test (e.g LDAP servers))
• Inventory files contain asset definitions, allowing the user to decide how the assets required by the tests are provided.
• Instances can be managed by proxmox-test-instance, e.g. automatically created, started and installed using the setup command and stopped/destroyed by running cleanup

Introduction

Goals

• Establish a framework that makes it easy to write integration tests targeting setups that are close to production setups
• Make it easy to run these tests against different kinds of setups (e.g. a 'blessed' set of test nodes in a CI or against development VMs provided by the developer, etc.)
• Provide great developer experience when running locally on a developer's workstation by being easy to set up and flexible enough to fit each developer's personal workflow. It should not feel like a chore to quickly run a set of integration tests before submitting patches.

Subcommands and global options

The runner provides the following subcommands:

• run <inventory> <testcases> - run a testcase file against the instances of an inventory file
• run-local <testcases> - run a testcase file on the local host, without an inventory
• hw-validation - run the installed hardware validation suite
• storage-plugin-validation <storage-id>... - validate one or more storages against the installed storage-plugin capability suite; see its --help output for the storage-specific parameters (migration targets, guest image, backup fallback storage, ISO URL)

run executes each test case's run command in the current working directory, so a suite that refers to its scripts as ./tests/... has to be started from the directory holding them. run-local and the installed-suite subcommands change into the directory of the testcase file first and can be started from anywhere.

All subcommands accept the global options --report <path> and --report-format text|json|junit|markdown to write a report file, --suite-version to record the suite version in the report for provenance, --verbose to print the entire output of each test, and --tags to select tests: a comma-separated list of tags replaces the suite's default selection, while entries prefixed with + add to it (e.g. --tags +storage-plugin-guest).

proxmox-test-scheduler

• Install proxmox-test-scheduler on the host that will coordinate the CI runs and create /etc/proxmox-test-scheduler/proxmox-test-scheduler.ron with at least the control_ip and pti_hosts entries, see example/proxmox-test-scheduler.ron. Note that control_ip is the address under which the test instances reach this host's package repository, so it must be routable from them.
• Every endpoint requires authentication, so create the first administrator before starting the daemon by writing /etc/proxmox-test-scheduler/init.toml (readable by the ptt user):

  admin-username = "admin@pts"
  admin-password = "..."
  

  On startup the daemon creates that account in the built-in pts realm and grants it the Admin role on /, unless it already exists.
• The scheduler needs to reach every host in pti_hosts as root over SSH. The systemd unit adds ssh_private_keyfile to an ssh-agent before starting the daemon; the matching public key has to be in root's authorized_keys on each test host.
• The ansible/ directory deploys all of the above - a scheduler plus any number of test hosts - see ansible/README.md. The daemon listens on HTTPS port 8443 and a background worker triggers runs on a configurable interval (scheduling_interval_secs).
• A run is created and then started via the API. The ticket endpoint returns the ticket as an HttpOnly cookie, and ticket authentication requires the CSRFPreventionToken header on every non-GET request:

  API=https://scheduler-host:8443/api2/json
  CSRF=$(curl -sk -c jar -d "username=admin@pts&password=..." \
    $API/access/ticket | jq -r '.data.CSRFPreventionToken')
  auth=(-b jar -H "CSRFPreventionToken: $CSRF")
  
  # create a run setup, returns a run id (rid)
  curl -k "${auth[@]}" -X POST $API/runs
  # upload packages under test for that rid (file content is the request body,
  # the file name is passed as the `file-name` parameter)
  curl -k "${auth[@]}" -X POST --data-binary @custom-package_1.0_amd64.deb \
    "$API/runs/<rid>/packages?file-name=custom-package_1.0_amd64.deb"
  # start the run
  curl -k "${auth[@]}" -X POST $API/runs/<rid>/start
  

  Starting a run rolls back or sets up instances, installs the uploaded packages, runs proxmox-test-runner, and emails the report if mail is configured. (All endpoints require the appropriate Test.* privileges.) API tokens (Authorization: PTSAPIToken=user@realm!name:secret, created via POST $API/access/users/<userid>/token/<name>) need no CSRF token and are the better fit for scripts.
• The setup plans reference installer ISOs from the PVE storage of the test host (local:iso/...), so prepare them there before the first run with proxmox-test-instance prepare-iso.

Test case declarations

Test cases are defined in RON files and passed to proxmox-test-runner as arguments.

In its most minimal form, a test case only needs a name and a command to execute (run). The command is executed in a shell, thus it is possible to run multiple commands or pass parameters the same way as you would in a shell script. Optionally, a timeout can be specified - if the script does not finish in that time, it will be killed by the test runner.

The commands specified by the run option are executed on the same host where the test runner is executed.

#![enable(implicit_some)]

TestcaseConfig(
    testcases: [
        Testcase(
            name: "test_cluster",
            run: "./pve_tests/test_cluster.pl",
            timeout: 60,
            instances: [
                "pve1",
                "pve2",
            ],
            // optional tag, used together with the runner's `--tags` option
            tag: "cluster",
        ),
    ],
    // optional: snapshot to roll back to between test cases
    rollback_between_tests: Some("pre-test"),
)

Test cases can also specify test instances via the instances key. A test instances is a resource needed by the test case, e.g. a PVE/PBS installation, or some other host with a specific setup needed by the test. Required instances are specified by their name (e.g. pve in the example above). In the example above, the test case requires access to an instance pve, which must be provided by the inventory file, otherwise the test is skipped.

When executing the test case, the instance configuration will be provided to the test case via a JSON file, the path to this JSON file will be provided via an environment variable ($CONFIG_PATH). The contents of the JSON file are as following:

{
  "schema_version": 1,
  "name": "pve_realm_basic",
  "instances": {
    "pve1": {
      "ip": "172.30.0.100",
      "root_password": "xxxxxxxx",
      "nodename": "pve1",
      "ssh_private_keyfile": "...",
      "virt": { "Qemu": { "vmid": 6000, "iso_type": "pve" } },
      "additional_disks": []
    }
  }
}

The virt field is required and selects the instance type: Qemu (with an iso_type of "pve", "pbs", ...) or Lxc. The Perl helpers dispatch on virt.Qemu.iso_type to decide whether get_pve_instance/get_pbs_instance returns. schema_version is asserted by the Perl framework so that an incompatible change to this format fails loudly instead of silently producing undefined fields.

Inside the test script, the contents can be accessed via jq or some other means. For test cases written in Perl, Proxmox/Test/Helper.pm and Proxmox/Test/PVEInstance.pm provide some useful abstractions that can be used to write test cases:

use Proxmox::Test::Helper;


# Reads the 'injected' JSON config from $CONFIG_PATH and returns
# a handle to the asset
my $pve1 = Helper::get_pve_instance('pve1');
# Use API client to make a request
my $res = $pve1->client()->get('/access/users/test@pve', {});

# Get SHA256 fingerprint of the nodes TLS cert
$pve1->get_fingerprint();
# Get host/IP of this asset
$pve1->get_ip();
# Get the root password
$pve1->get_root_password();
# Get host/IP of this asset
$pve1->get_nodename();

# Helper that waits for a task to finish
$pve1->wait_for_task($upid);

Process and signal behavior

The test runner does not own any test assets, so it installs no signal handlers: on SIGINT/SIGTERM it is terminated by the default disposition. Asset lifecycle (creation, snapshots, rollback, cleanup) is managed by proxmox-test-instance, not the runner.

Each test command runs in its own process group. If a test exceeds its timeout, the runner signals the whole group (SIGTERM, then SIGKILL after a short grace period) so that child processes spawned by the test (for example ssh or ansible) are terminated as well, and reaps the process to avoid zombies.

TODO

Some future improvements, without any ranking:

• Capture systemd journal for test assets (controllable via an option in asset config?)
• Implement an interactive login command, allowing one to use the test runner without using a token
• Create token for test host on-demand, with minimal privileges
• Introduce provides/requires mechanism with for test assets, allowing to specify additional requirements for test assets (e.g. a certain storage type, PVE version, etc.).