Behat

Vortex uses Behat for Behavior-Driven Development (BDD) testing. This page is for the developers who write and run those tests. Follow it to run any feature or tag, use the shipped steps, find the screenshots and see how continuous integration (CI) splits and retries your scenarios. For unit, kernel and functional tests of PHP code, see PHPUnit.

You describe how the site should behave in stories written in the Gherkin language. Behat runs every step of each story against the provisioned site. These are end-to-end tests, so spend them on your critical user journeys, such as logging in or resetting a password.

Vortex includes a pre-configured behat.php with profiles for Drupal projects, and a browser container running Chrome, which you can watch through noVNC while the tests run.

Usage

Behat needs a provisioned Drupal site, so run it inside the container environment with the following commands:

Ahoy

# Run all Behat tests
ahoy test-bdd
# Run specific feature file
ahoy test-bdd tests/behat/features/homepage.feature
# Run scenarios with specific tag
ahoy test-bdd -- --tags=@smoke

Docker Compose

# Run all Behat tests. The `-d memory_limit=-1` flag lifts the PHP
# memory limit for long runs, matching what `ahoy test-bdd` does.
docker compose exec cli php -d memory_limit=-1 vendor/bin/behat
# Run specific feature file
docker compose exec cli php -d memory_limit=-1 vendor/bin/behat \
  tests/behat/features/homepage.feature
# Run scenarios with specific tag
docker compose exec cli php -d memory_limit=-1 vendor/bin/behat \
  --tags=@smoke

The -- before Behat's options is optional: Ahoy drops it and passes the rest of the line to Behat. See Ahoy > Usage.

Discover available step definitions

Behat Steps, Mink and Behat Screenshot each bring their own steps. To list every step definition your suite can use, run the following command:

Ahoy

# Generate step definitions reference
ahoy test-bdd -- --definitions=l

Docker Compose

# Generate step definitions reference
docker compose exec cli php -d memory_limit=-1 vendor/bin/behat \
  --definitions=l

Write tests

Put each feature file in the tests/behat/features directory, where the default suite finds it, and structure it like the following example:

@tag_describing_feature_group
Feature: Short feature description

  As a site owner
  I want to make sure that my feature does what it is supposed to do
  So that I can be sure that my site is working as expected

  Scenario: Anonymous user uses a feature
    Given I go to the homepage
    Then the path should be "<front>"
    And I save screenshot

  @javascript
  Scenario: Anonymous user uses a feature with AJAX
    Given I go to the homepage
    When I click on the element ".button"
    Then I save screenshot

Untagged scenarios run through an HTTP client that doesn't execute JavaScript. To run a scenario in a real browser, tag it @javascript: the scenario then runs in Chrome in the chrome container, driven through Selenium. A @javascript scenario runs slower than an untagged scenario, so add the tag only when a scenario needs JavaScript.

Skip tests

To exclude a feature or scenario from the test run, add the @skipped tag to it. Every profile in behat.php filters it out.

Boilerplate features

Vortex provides the following boilerplate features:

• Core user journeys: homepage, login, password reset and search
• The shipped module and service integrations: Redis, ClamAV, redirects, robots.txt, XML sitemap and accessibility
• behat.feature, which checks that Behat itself is configured correctly
• The demo features counter.feature and pages.feature, tagged @demo

The installer removes the features for the services and modules you don't select. The homepage, login and Behat configuration features are tagged @smoke.

These boilerplate tests run in the continuous integration pipeline from the moment you install Vortex. They make a good starting point for your own.

Email

To capture outgoing messages instead of delivering them, tag a scenario @email. Then assert on their recipients and content with the steps from EmailTrait. The password reset feature shows the pattern.

To capture the messages that a module sends through its own mail interface, tag the scenario @email:TYPE, where TYPE is the interface's key in the system.mail configuration. A scenario with only typed tags collects only those interfaces, so add @email:default to collect the default interface too.

In the ci environment, settings.system.php makes the same mail collector the default backend for every message sent through interface.default. CI therefore stores those messages instead of delivering them, even when a scenario isn't tagged. The collector is a configuration override in a settings file, so a configuration import or a change in the UI can't switch it off. Reroute Email is disabled in ci, so each stored message keeps its intended recipient.

The collector covers interface.default, the only interface the shipped configuration defines. Sites that use Symfony Mailer or Mail System can route a module through its own interface: interface.<module> or interface.<module>_<key>. Outside scenarios tagged @email:TYPE, messages sent through those interfaces bypass the collector. For that case, docker-compose.yml also points SSMTP_MAILHUB at a closed port in CI, so mail sent through sSMTP fails at the transport. A custom SMTP or API transport bypasses sSMTP, so give each custom interface a transport that doesn't deliver from CI.

The bare When the test email system queue is cleared step still needs the @email tag or an earlier When the test email system is enabled step. Each tagged scenario also clears the queue as it starts, so messages from an earlier scenario never reach your assertions.

Project conventions

Your project's own Behat conventions live in its docs/testing.md file, which your project team maintains to record the testing practices you agree on. It covers the following topics:

• User story format
• Standard user types
• Test data
• An example feature file
• The process for testing a content type

Vortex ships the file in the project's docs/ directory. The installer keeps it unless you decline its Preserve project documentation? prompt. Declining the prompt removes the whole docs/ directory.

Configuration

See the Behat user guide and its configuration reference.

All global configuration takes place in the behat.php file, because Behat 4 reads its configuration only from PHP.

By default, Behat runs all tests defined in the tests/behat/features directory.

To add or remove test targets, change the suite's paths:

behat.php

$suite = (new Suite('default'))
  ->withPaths('%paths.base%/tests/behat/features', '%paths.base%/web/modules/custom/mymodule/tests/behat/features')

Chrome session flags

The behat.php configuration passes the following flags to Chrome through goog:chromeOptions, so the tests run the same way every time in the container environment:

Flag	Purpose
--disable-extensions	Prevents interference from browser extensions
--disable-popup-blocking	Allows tests to open popups without being blocked
--disable-translate	Prevents the translation bar from appearing on non-English pages
--force-prefers-reduced-motion	Disables CSS animations and transitions for test stability
--test-type	Suppresses error dialogs and crash recovery prompts
--window-size=1920,1080	Sets a deterministic viewport size for consistent screenshots

--window-size only sets the size Chrome opens at. The Behat Screenshot context resizes the window to 1440 by 900 pixels at the start of every @javascript scenario.

Contexts and extensions

The configuration uses the following contexts and extensions:

• Behat Steps: a library of reusable Behat steps for Drupal. It also supplies the Drupal integration layer as the DrevOps\BehatSteps\Extension\ServiceContainer\BehatStepsExtension extension. behat.php configures it with the Drupal API, Drush and black-box backends, the theme's region map and the message and login selectors.
• Mink Extension: the browser sessions and the base browser vocabulary, registered as the Behat\MinkExtension\ServiceContainer\MinkExtension extension and the Behat\MinkExtension\Context\MinkContext context. Untagged scenarios use a BrowserKit session that sends HTTP requests from PHP. @javascript scenarios use the selenium2 session, which drives Chrome at http://chrome:4444/wd/hub.
• Behat Screenshot: the extension and steps that capture HTML and image screenshots on demand or when a test fails, registered as the DrevOps\BehatScreenshotExtension\ServiceContainer\BehatScreenshotExtension extension and the DrevOps\BehatScreenshotExtension\Context\ScreenshotContext context.

FeatureContext

The FeatureContext.php file is your project's own Behat context. The default suite loads it, and it's where your custom step definitions and hooks go. Behat 4 reads them from PHP attributes such as #[Given] and #[BeforeScenario]. It ignores docblock annotations such as @Given.

FeatureContext extends DrevOps\BehatSteps\DrupalContext from the Behat Steps package. That context already includes every step the package provides, so the file holds only the project's own step definitions and hooks. Don't add the package's WebContext to the suite. Registering it beside DrupalContext defines every web step twice, and Behat stops with a fatal error.

Reports

Behat writes test run reports in JUnit format to the .logs/test_results/behat directory.

Screenshots

Behat Screenshot saves a screenshot whenever a step fails, and whenever a step such as I save screenshot runs. Each capture is an HTML file of the page, headed by its URL, feature, step and time, plus a full-page PNG image for @javascript scenarios. The HTTP client behind untagged scenarios can't take images, so those get the HTML file only. To capture a screenshot after every passed step of a scenario, tag the scenario or its feature @screenshots.

Screenshots land in the .logs/screenshots directory by default. Screenshots from earlier runs stay there, because behat.php sets purge to FALSE. To change either behavior, set the matching environment variable inside the cli container:

• BEHAT_SCREENSHOT_DIR stores the screenshots in another directory.
• BEHAT_SCREENSHOT_PURGE set to 1 empties the directory at the start of a run.

ahoy test-bdd doesn't pass these variables through from your host, so set them with docker compose exec -e, as the CI step does. For example, the following command empties the directory, then runs Behat:

docker compose exec -e BEHAT_SCREENSHOT_PURGE=1 cli \
  php -d memory_limit=-1 vendor/bin/behat

Animated screenshots

Behat Screenshot combines the per-step screenshots into an animated GIF for each scenario, so a failing test is easier to follow. behat.php turns this on under the animation key of the BehatScreenshotExtension extension, with 500 milliseconds between frames (frame_delay). The frames are the PNG images, so only @javascript scenarios produce a GIF.

Animation captures a screenshot after every passed step. With always_fullscreen on, each capture covers the full page. The cost grows with the number of steps and roughly doubles the wall time of a run. To turn animation off everywhere, set animation.enabled to FALSE in behat.php. To exempt a single scenario or feature, tag it @screenshots:animated:skip.

With animation off, the @screenshots:animated tag opts a single scenario or feature back in. A tag on a scenario overrides a tag on its feature. See Artifacts for how the continuous integration pipeline handles animation.

Output format

By default, Vortex uses Behat's built-in progress formatter with its inlineFailures option turned on in behat.php. It prints a dot for each passed step and the details of each failed, pending or undefined step as soon as the step runs. Without the option, Behat prints those details only in the summary at the end of the run. The test run continues after a failure while the output stays minimal.

Watch the browser

For @javascript tests, Behat drives Chrome through Selenium in the chrome container. That container also runs a VNC server with noVNC, so you can watch the browser from a browser on your host machine. To open it, follow the steps for your setup:

Ahoy

1. Run ahoy info to print the project information.
2. Open the URL on the Selenium VNC URL on host line in a browser on your host machine. noVNC connects with the password secret.

Docker Compose

1. Run docker compose port chrome 7900 to print the host port that maps to the container's noVNC port, 7900.
2. Open http://localhost:PORT/?autoconnect=1&password=secret in a browser on your host machine. Replace PORT with the number after the colon in the output of step 1.

Confirm that noVNC shows the Chrome window: run ahoy test-bdd tests/behat/features/homepage.feature, and watch Chrome open the site's homepage during the feature's @javascript scenario.

To work with the page yourself in the middle of a test, add an And I wait for 60 seconds step to the scenario. The step pauses the test for a minute, so you can use the browser in the container through noVNC on your host machine.

Continuous integration

In the continuous integration pipeline, Behat runs with --strict, so an undefined or pending step fails the run there. ahoy test-bdd runs without --strict, so to match CI locally, run ahoy test-bdd -- --strict.

Profiles

Without a --profile option, Behat uses the default profile defined in behat.php: it runs every scenario except those tagged @skipped.

When the pipeline has more than 1 runner, Behat tests run in parallel across the runners, so the suite finishes sooner. Tags mark which runner a feature or scenario lands on. The CI step picks the profile named after the runner index, starting from p0. Vortex supports any number of parallel runners but ships 2 parallel profiles, p0 and p1, to match the 2 runners the pipeline starts with:

• p0, on the first runner, is the catch-all: it runs every scenario not tagged @p1, plus all @smoke scenarios.
• p1, on the second runner, runs scenarios tagged @p1, plus all @smoke scenarios.

Both profiles skip anything tagged @skipped. In practice, leave a feature untagged to run it on the first runner. Tag it @p1 to move it to the second runner, or tag it @smoke to run it on every runner. A feature you forget to tag still runs on the first runner.

In the following example, with 1 runner, Behat runs both scenarios. With 2 runners, the first scenario runs on the first runner (untagged scenarios stay there) and the second scenario runs on the second runner:

@homepage
Feature: Homepage

  Ensure that homepage is displayed as expected.

  Scenario: Anonymous user visits homepage
    Given I go to the homepage
    Then the path should be "<front>"
    And I save screenshot

  @javascript @p1
  Scenario: Anonymous user visits homepage
    Given I go to the homepage
    Then the path should be "<front>"
    And I save screenshot

To add more p* profiles, copy the existing p1 profile in your behat.php, change its name and tag filter, and register it in the returned Config. Then exclude the new tag from the p0 catch-all, or its scenarios run on the first runner as well. See Adding more containers for the full steps.

Profile numbers count from the first runner that runs Behat. The CI configuration subtracts VORTEX_CI_BEHAT_PROFILE_OFFSET (0 by default) from the runner index to pick the profile. As a result, you can dedicate leading runners to other tools and keep your profile names and feature tags. See Test parallelism for the container mechanics.

To force a profile, set VORTEX_CI_BEHAT_PROFILE, as a repository variable in GitHub Actions or an environment variable in CircleCI. Every runner then uses that profile. On a pipeline with 1 runner, Behat uses the default profile unless VORTEX_CI_BEHAT_PROFILE is set.

Artifacts

The continuous integration pipeline stores the test results and screenshots as artifacts. Use the JUnit reports to track test performance and stability. In GitHub Actions, each runner uploads its .logs directory as the test-artifacts-<runner index> artifact, which you download from the run's Summary page. Unless VORTEX_CI_TEST_RESULTS_SKIP is 1, the JUnit reports also appear among the run's checks as Test results (<runner index>). In CircleCI, the screenshots are in the job's Artifacts tab and the JUnit reports are in its test results.

In continuous integration, the Behat step passes BEHAT_SCREENSHOT_ANIMATION_SKIP=1 to skip animation, while local runs, where a recording is worth the wait, keep animation. The variable overrides both behat.php and the animation tags.

Flaky tests

A Behat scenario can occasionally fail without a real bug behind it, on a slow page load or a timing-sensitive step. How often depends on the size of your site and the resources of the continuous integration runner.

The CI step therefore retries once: when the first run fails, the step runs Behat again with --rerun, which repeats only the failed scenarios. The step fails only when a scenario fails in both runs. If the first run failed without recording a failed scenario, --rerun runs the whole profile again.

A scenario that passes only on the second run is still flaky, so check the first run's output in the job log.

Ignore failures

Set VORTEX_CI_BEHAT_IGNORE_FAILURE to 1 to let the build pass despite failing scenarios. Behat still runs, retries as described in Flaky tests and reports every failure in the job log. See Ignore tool failures.