epic: OpenSSH drop-in compatibility, measured against the OpenSSH regress suite

[inureyes]

Goal

Make bssh a true drop-in replacement for OpenSSH ssh(1) in single-destination mode, with bssh's parallel, cluster, and pdsh features layered strictly on top as additive extensions. OpenSSH semantics become the substrate; nothing bssh adds may take a name or a behavior that OpenSSH already defines.

Conformance is measured against the OpenSSH project's own regression suite (regress/ in openssh-portable), not against our own reading of the manual. The suite is the instrument, and its score is the acceptance gate.

Measured baseline (bssh 2.4.3)

Measurement performed on 2026-08-26 against a self-built openssh-portable V_10_3_P1 on macOS arm64, using 89 client-relevant tests selected from the suite's 116 shell tests. Every bssh failure was re-run with the stock OpenSSH client to separate genuine incompatibilities from environment limitations.

Run	Result
bssh, unmodified	0 / 89
bssh, behind a shim that absorbs -E	23 PASS / 10 SKIP / 56 FAIL
OpenSSH baseline re-run of those 56 failures	48 baseline-PASS, 8 baseline-FAIL

Score on environment-valid, non-skipped tests: 23 / 71 (32%).

The unmodified run scored zero because test-exec.sh wrapped every client invocation as exec ${SSH} -E${logfile} "$@", and bssh rejected -E. One missing flag made the entire suite unrunnable.

Tests that passed in that baseline: addrmatch, agent-subprocess, agent-timeout, broken-pipe, cfgparse, dhgex, exit-status-signal, keygen-knownhosts, keyscan, krl, proto-mismatch, proto-version, scp, scp-uri, servcfginclude, sftp, sftp-badcmds, sftp-batch, sftp-cmds, sftp-glob, sftp-perm, sftp-resume, sshsig.

Root causes behind the 48 genuine failures

1. The ssh_config layer parses far more than it uses. Of 81 fields on the resolved host config, only about 10 are read anywhere outside src/ssh/ssh_config/: hostname, user, port, identity files, StrictHostKeyChecking, ProxyJump, Compression, AddressFamily, ServerAliveInterval and ServerAliveCountMax. The rest are parsed, validated, merged by the resolver, and then never consulted.
2. Host key file selection is the largest single cluster. src/ssh/known_hosts.rs:20 always resolves to ~/.ssh/known_hosts; user_known_hosts_file and global_known_hosts_file are assigned in resolver.rs:120-124 and never read. bssh also validates those paths strictly enough to reject GlobalKnownHostsFile /dev/null, a standard OpenSSH idiom, as a "security violation" that aborts config parsing outright.
3. Missing flags, ranked by how often the suite uses them. Across 1074 traced client invocations: -E 1071, -G 104, -c 84, -m 42, -s 8, -n 8, -W 4, -S 3, -O 2, -M 1.
4. Short flags that collide with OpenSSH. -N, -f, -C, -A, -S, -k and -b all mean something different in bssh than in ssh(1). dynamic-forward.sh uses four of them in a single invocation.
5. Output is not byte-transparent. bssh writes a decorative banner and a summary rule to stderr even for a single host, emits ANSI escapes when stderr is not a terminal, ignores NO_COLOR, and routes -V to stdout where OpenSSH uses stderr.
6. Diagnostics are opaque. 20 of the 48 failures surface only as Error: I/O error, which covers connection refusal, authentication failure, and channel errors alike.

Governing rule for the superset

In a single-destination invocation, OpenSSH semantics win unconditionally. bssh extensions are reachable only through long flags. The pdsh single-letter meanings survive only under --pdsh-compat or when invoked as pdsh via argv[0]. This is the only rule under which "fully OpenSSH compatible, plus more" is self-consistent, and it makes the flag reassignment in the last sub-issue a breaking change scheduled for 3.0.

Explicitly out of scope

These are recorded as permanent skips rather than pending work, and the harness must report them as such: GSSAPI authentication, PKCS#11 and FIDO/security-key middleware, X11 forwarding, tun device forwarding (-w), KRL generation, certificate authority key signing, sshd-side configuration tests, and interoperability with PuTTY, Dropbear, Twisted conch and ssh.com.

Sub-issues

Ordered by leverage measured against the suite, not by size.

Phase 1

• test: add an OpenSSH regress compatibility harness and score gate #276 Add an OpenSSH regress compatibility harness and score gate

Phase 2

• feat(cli): implement -E log_file #277 Implement -E log_file

Phase 3

• fix(ssh_config): honor UserKnownHostsFile and GlobalKnownHostsFile #278 Honor UserKnownHostsFile and GlobalKnownHostsFile
• fix(ssh_config): execute ProxyCommand instead of parsing and discarding it #280 Execute ProxyCommand instead of parsing and discarding it
• fix(ui): make single-host output byte-transparent and honor NO_COLOR #283 Make single-host output byte-transparent and honor NO_COLOR

Phase 4

• fix(ssh_config): honor HostKeyAlias when looking up host keys #279 Honor HostKeyAlias when looking up host keys (depends on fix(ssh_config): honor UserKnownHostsFile and GlobalKnownHostsFile #278)

Phase 5

• feat(ssh_config): add directive registry and transport runtime foundation #295 Add the ssh_config runtime foundation (depends on fix(ssh_config): honor HostKeyAlias when looking up host keys #279, fix(ssh_config): execute ProxyCommand instead of parsing and discarding it #280)
• fix(error): replace the opaque I/O error with actionable diagnostics #284 Replace the opaque I/O error with actionable diagnostics (depends on fix(ssh_config): honor HostKeyAlias when looking up host keys #279, fix(ssh_config): execute ProxyCommand instead of parsing and discarding it #280, fix(ui): make single-host output byte-transparent and honor NO_COLOR #283)

Phase 6

• feat(ssh): enforce ssh_config authentication policy and user certificates #296 Enforce ssh_config authentication policy and user certificates (depends on feat(ssh_config): add directive registry and transport runtime foundation #295)
• feat(ssh): apply ssh_config environment, command, TTY, and session policy #297 Apply ssh_config environment, command, TTY, and session policy (depends on feat(ssh_config): add directive registry and transport runtime foundation #295)
• feat(ssh): enforce known-host policy, DNS checks, and secure key rotation #299 Enforce known-host policy, DNS checks, and secure key rotation (depends on feat(ssh_config): add directive registry and transport runtime foundation #295)
• feat(ssh): enforce RekeyLimit through russh transport limits #301 Enforce RekeyLimit through russh transport limits (depends on feat(ssh_config): add directive registry and transport runtime foundation #295)

Phase 7

• feat(ssh): apply ssh_config forwarding directives to live connections #298 Apply ssh_config forwarding directives to live connections (depends on feat(ssh_config): add directive registry and transport runtime foundation #295, feat(ssh): apply ssh_config environment, command, TTY, and session policy #297)
• feat(net): honor ssh_config source binding and IPQoS #300 Honor ssh_config source binding and IPQoS (depends on feat(ssh_config): add directive registry and transport runtime foundation #295, feat(ssh): apply ssh_config environment, command, TTY, and session policy #297)

Phase 8

• feat(ssh_config): wire the parsed-but-unused ssh_config keywords #281 Finalize all parsed-but-unused ssh_config keywords (depends on feat(ssh_config): add directive registry and transport runtime foundation #295, feat(ssh): enforce ssh_config authentication policy and user certificates #296, feat(ssh): apply ssh_config environment, command, TTY, and session policy #297, feat(ssh): apply ssh_config forwarding directives to live connections #298, feat(ssh): enforce known-host policy, DNS checks, and secure key rotation #299, feat(net): honor ssh_config source binding and IPQoS #300, feat(ssh): enforce RekeyLimit through russh transport limits #301); implementation children are merged, and remaining compatibility gaps continue in feat(ssh_config): wire the parsed-but-unused ssh_config keywords #281 and fix(openssh): close residual regress compatibility gaps #314

Phase 9

• feat(cli): implement -G to dump the resolved configuration #282 Implement -G to dump the resolved configuration (depends on feat(ssh_config): wire the parsed-but-unused ssh_config keywords #281)

Phase 10

• feat(cli): implement -c, -m, -s, -n and -W #285 Implement -c, -m, -s, -n and -W (depends on feat(cli): implement -G to dump the resolved configuration #282)

Phase 11

• feat(ssh): support connection multiplexing (-M, -O, ControlPath) #286 Support connection multiplexing (-M, -O, ControlPath) (depends on feat(cli): implement -c, -m, -s, -n and -W #285)

Phase 12

• fix(cli)!: give OpenSSH back its short flags (-N, -f, -C, -A, -S, -k, -b) #287 Give OpenSSH back its short flags (breaking, 3.0) (depends on fix(error): replace the opaque I/O error with actionable diagnostics #284, feat(ssh): support connection multiplexing (-M, -O, ControlPath) #286)

Final result (2026-09-01)

PR #313 completed the final integration. The pinned OpenSSH V_10_3_P1 harness reports:

• macOS: 60 passed out of 66 eligible tests, 6 failed, 7 environmental, 6 skipped
• Ubuntu: 61 passed out of 69 eligible tests, 8 failed, 4 environmental, 6 skipped

Both platforms meet the 60-pass gate. The remaining candidate-only failures are enumerated with platform evidence in #314. The broader parsed-keyword tracker #281 remains open for residual compatibility work; neither issue blocks completion of this score-gated epic.

The breaking short-flag migration cannot retroactively add a warning cycle to an already published 2.x release. The merged implementation emits bounded interactive migration warnings, documents the long-option replacements, and keeps redirected protocol streams and -E logs byte-transparent.

Acceptance criteria for the epic

• The regress harness runs in CI and reports a pass/skip/fail score against a committed baseline.
• bssh scores at least 60 of the 71 environment-valid, non-skipped tests, with every remaining failure either listed in the out-of-scope set above or tracked by an open issue.
• README.md no longer claims unqualified drop-in replacement status; it links to the measured score and the documented skip list.
• Every keyword bssh's ssh_config parser accepts is either consumed at runtime or rejected with a warning that says it is unimplemented. No keyword is silently parsed and discarded.