You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(site-command): add a Behat suite and CI workflow - #518
Extends the three old feature files into a Behat suite for site-command (features/: 211 scenarios, outline rows counted) and replaces test_and_build.yml with a Behat workflow that runs it on every pull request, on pushes to develop and master, and on demand.
It covers create, delete, list and info, routing, enable and disable, restart and reload, clean and the proxy cache, SSL (self-signed, custom, inherited, Let's Encrypt issuance, renewal and aliases), alias, SSL and PHP updates, www redirects, share, backup and restore (local and remote), and clone and sync. features/COVERAGE.md maps each command, flag and behaviour to its scenarios.
The workflow runs the suite inside a fresh core checkout: core develop is cloned, every easyengine/* package is set to dev-develop, and this package is installed from the pull request's checkout through a Composer path repository, so the code under test is the PR's. The old job tested the Packagist dev-develop package instead.
Lanes: main (190 scenarios), wp (10, WordPress sites) and le (11), each on PHP 7.4 and 8.0 to 8.5. The main lane's jobs are named Behat Tests - PHP <version>, so the required checks keep matching. The main lane also builds the phar without dev autoloading, as the old job did. @destructive scenarios run there too, since the runner is thrown away.
@le runs against a local Pebble ACME server (.github/behat/pebble-setup.sh). Every other lane points the Let's Encrypt host at a blackhole and checks that the real service can't be reached, and on CI the suite refuses to start unless one of the two is in place, so no run can reach the real Let's Encrypt.
@remote backup scenarios use a local rclone alias remote.
Scenarios written for open bugs (24) are left out for now: they assert the fixed behaviour, so each comes back with its fix. Every scenario in this PR gates, and COVERAGE.md lists the bugs that aren't covered yet.
To run it locally (as root, on a test machine, since the suite creates and deletes *.test sites): copy features/ over a core checkout's features/ and run sudo -E vendor/bin/behat --tags='~@le&&~@remote' from the core root; add ~@destructive to skip the scenarios that change global services or config.
Rework the context the site-type packages copy:
- delete only sites matching SITE_PREFIX after each feature (never every listed site), purge the prefix's remote backups for @Remote features, and undo per-scenario hosts entries, scratch dirs and backup leftovers
- anchor the old regex steps: the unanchored `I run '(.*)'|"(.*)"` matched any step with a quoted string, and the `:index` placeholders inside regexes never matched
- fix the inverted container label count check and look containers up by label instead of v1 compose names
- add generic steps for exit codes, output, JSON, the sites table, files, hosts entries, docker objects, HTTP (resolved to 127.0.0.1, the public IP or verified DNS), TLS certificates, polling waits, config guards and rclone backups, with {NAME} variables in every argument
…vers
Purge a site's remote backups only after listing the rclone path and
finding that exact directory, and fail if backups remain. After each
feature, once no suite site is left, remove suite-named proxy certs,
ACME state, site dirs and volumes that known bugs leave behind. Add a
guard step for host-wide LE commands and keep /etc/hosts lines intact
when the file lacks a trailing newline.
- Hold the fake backup lock with flock(1) so the scenario also works
once PR EasyEngine#480 switches to flock, and assert a failed restore through
the next backup instead of the lock file.
- Use a reserved .invalid name for the failing LE alias instead of a
real rt.gw host outside the test subdomain.
- Guard ssl-renew --all against other LE sites on the host.
- Poll after reloading custom nginx config instead of one request.
- Check SEC-1 leftovers by label and directory.
- Add known-bug scenarios for update --php on html (SC-13) and
update --ssl=custom (SC-15), and reference SC-* IDs and the open
PRs that fix each known bug.
Correct the tag counts, document the remote listing, leftover cleanup
and LE guards, map every known-bug scenario to its issues.md ID and the
open PR that fixes it, and list the PRs that change what passing
scenarios assert.
Table output needs SHELL_PIPE=0 without a TTY, and JSON is pretty-printed. The clone error scenario needs a site on the host (SC-17). The self-signed ssl-info DNS scenario (SC-16) and the invalid --proxy-cache value (SC-18) expose product bugs, so they're tagged @known-bug; a no-SSL site now covers the non-DNS branch.
Backup ids have one-second resolution (BKP-5). On a fast local remote two backups landed in the same directory, so the retention scenario passed without testing anything and the by-id restore scenarios could compare a backup with itself.
On PHP 7.4 is_file() answers from the stat cache for the last path checked, so a file ee deleted in between still exists: "Deleting a self-signed site removes its certificate" and both redirect removal scenarios failed on the 7.4 runner, and the MIG-15 known-bug scenario passed vacuously there.
…gine#502)
AUTH-1 is fixed on develop, so the underscore alias scenario is a regular regression check. "Delete an alias domain" now waits for the remaining alias to answer 200 before checking the deleted one: the recreated nginx container drops all of the site's domains from the proxy until docker-gen catches up, and on develop the command returns inside that gap.
…ct collision
An html site whose name maps to the volumes and compose project of an existing site (bh-site-command-160x.test vs bh-site-command-160.x.test) is refused and leaves the existing site untouched. Needs the fix/site-name-collisions branch of site-command.
Regular now (fail on develop until fix/proxy-cache-rollback-and-le-alias merges):
- SSL-2: a failing LE alias is rejected, the DB, certificate and renewals are unchanged.
- SSL-5/SSL-9: deleting an LE site leaves no ACME state.
- SC-3: `--ssl=off` works on a self-signed site; it also removes the certificate and the redirect's HTTPS block (WPT-12).
New: deleting a site without SSL removes a leftover certificate, `--ssl=off` on an LE site removes the certificate and ACME state, deleting an alias removes its proxy cache location, and a failed proxy cache alias sync (PRX-6) leaves a config nginx can test and restart with.
…d restore scope
Known-bug scenarios for BKP-7 (invalid no-of-backups), BKP-8 (hosts sharing rclone-path, via two backup-host-id config copies), BKP-9 (WordPress webroot files), BKP-10 (symlinked uploads), BKP-12 (restore without backups) and BKP-14 (dotfiles left by a restore). The remote-backup steps now also find backups under <rclone-path>/<site>/<host-id>/.
…ecks
Adds @known-bug scenarios for CLONE-3, CLONE-7 and CLONE-8 (fixed on fix/clone-sync-safety): a loopback address as the same host, ssh options in a remote location, a remote `.`, a site name with shell syntax, the sync help, --ssl=custom without --ssl-crt, and admin passwords with shell syntax or characters the destination shell would expand.
… regular ones
WP-1, SC-3, SC-15, CACHE-1, CLONE-3/7/8, SEC-1 and SEC-3 pass on develop. The wildcard-update refusal is checked with --ssl=le, since self-signed certificates are always wildcard (5c34be8).
…line output
compareContents() compared list elements by position, so [1,3,5] wasn't found in [1,2,3,4,5], and a missing key or property raised a notice. Both something-like steps only used the first line of the expected text.
Runs the package's features/ inside a fresh core develop checkout, with this checkout installed as the package through a Composer path repository, so the PR's own code is under test. @le is excluded (no public DNS on runners); @Remote backups use a runner-local rclone alias remote. @known-bug scenarios run in a separate non-gating job.
EE's hardcoded Let's Encrypt endpoint is redirected to a local Pebble on the runner (hosts entry, throwaway CA, resolved routing for the test domain). No product code changes.
Run PHP 7.4 and 8.5, strip control characters from the junit files before building failures.txt, and cap the Behat step at 75 minutes so a hung run still uploads its logs. The @le lanes against Pebble move into behat.yml (lanes named le*), replacing behat-le.yml. Diagnostics: the host addresses and routes, and the docker-compose version after the first run (MIG-20).
EE's first run downloads docker-compose 2.27.0 without --fail (MIG-10); on a GitHub error it moves the HTML error page over the binary, as on an admin-tools-command runner. Fetch 2.27.0 again with --fail and retries in that case.
…uest or query fails
A negative HTTP check now needs a real response: a transport error fails the one-shot checks and keeps the wait step polling. Docker helpers check the exit status before reading output, and named-object absence comes from a successful ls instead of an inspect failure.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unresolved test reliability issues, the unmerged core prerequisite, and privileged network and trust-store changes require fixes and final human validation.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Assert WP-CLI preparation and verification commands succeed
features/clone-sync.feature:138
The negative output assertion passes when wp option get blogname fails with empty STDOUT, so a files-only sync that breaks the destination database can pass this scenario. The source wp option update result is also hidden by the subsequent sync command. Assert that both WP-CLI commands succeed before checking their effects, so failed preparation or a failed database query cannot produce a passing test.
Wait for certificate to stop covering the removed alias
features/le.feature:75
Waiting for HTTP 200 does not wait for the replacement certificate. The old certificate still validates {LE1}, so both the wait and the verified GET can succeed while nginx serves the certificate containing a1.{LE1}. The immediate negative coverage assertion can then fail during a normal asynchronous proxy reload. Add a bounded negative certificate-coverage polling step and use it here, retrying until the served certificate no longer covers the removed alias.
Test does not cover deleting a site with an attached alias
features/le.feature:156
On a fresh host, the site "{LE1}" does not exist does nothing, so this scenario passes without creating or deleting any ACME state. In the full feature, the preceding scenario deletes LE1 only after a1.{LE1} has been removed from its aliases, so this also does not test deletion of a site with an attached alias. Arrange a site with an alias, confirm that its www and alias ACME directories exist, then delete it and check their removal in the same scenario. This can be combined with the preceding deletion scenario to avoid a separate fixture.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Root-level DNS, firewall, and trust-store changes warrant human validation, and a local-host test assertion remains unresolved.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Avoid asserting shared ngrok binary was initially absent
features/share.feature:21
This assertion fails on a local test host where an earlier ee site share installed ngrok, even when the SSL rejection works correctly. The binary is shared across sites, and this scenario never establishes that it was absent before the command. Remove the unconditional absence check, or record the binary's initial state and verify that it remains unchanged. Do not delete a pre-existing global binary as test setup.
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the three old feature files into a Behat suite for site-command (
features/: 211 scenarios, outline rows counted) and replacestest_and_build.ymlwith aBehatworkflow that runs it on every pull request, on pushes todevelopandmaster, and on demand.It covers create, delete, list and info, routing, enable and disable, restart and reload, clean and the proxy cache, SSL (self-signed, custom, inherited, Let's Encrypt issuance, renewal and aliases), alias, SSL and PHP updates, www redirects, share, backup and restore (local and remote), and clone and sync.
features/COVERAGE.mdmaps each command, flag and behaviour to its scenarios.developis cloned, everyeasyengine/*package is set todev-develop, and this package is installed from the pull request's checkout through a Composer path repository, so the code under test is the PR's. The old job tested the Packagistdev-developpackage instead.Behat Tests - PHP <version>, so the required checks keep matching. The main lane also builds the phar without dev autoloading, as the old job did.@destructivescenarios run there too, since the runner is thrown away.@leruns against a local Pebble ACME server (.github/behat/pebble-setup.sh). Every other lane points the Let's Encrypt host at a blackhole and checks that the real service can't be reached, and on CI the suite refuses to start unless one of the two is in place, so no run can reach the real Let's Encrypt.@remotebackup scenarios use a local rclone alias remote.Merge after EasyEngine/easyengine#1950, which stops core's test job from running this suite unfiltered.
To run it locally (as root, on a test machine, since the suite creates and deletes
*.testsites): copyfeatures/over a core checkout'sfeatures/and runsudo -E vendor/bin/behat --tags='~@le&&~@remote'from the core root; add~@destructiveto skip the scenarios that change global services or config.Green run on my fork (PHP 7.4 and 8.0 to 8.5, every lane): https://github.com/mrrobot47/site-command/actions/runs/39377584767