Swift: Convert .qlref security query tests to inline expectation tests

[Copilot]

Converts 19 Swift security query .qlref tests from .expected-file-based testing to inline expectation tests by adding // $ Alert, // $ Source, and // $ Sink comments directly in the Swift source files.

What changed

• 19 .qlref files updated to add postprocess: utils/test/InlineExpectationsTestQuery.ql (compact single-line form)
• 35 Swift source files annotated with inline expectation comments derived from existing .expected files:
  • @kind problem queries: // $ Alert at each result's end-line
  • @kind path-problem queries: // $ Alert at alert location, // $ Source at source (when on different line), // $ Sink at sink (when on different line from alert)
• No .expected files were modified

Before/after example (CWE-760/ConstantSalt):

// Before
let myConstantSalt1 = Data("abcdef123456")  // no annotation
let _ = myEncryptor.key(..., salt: myConstantSalt1, ...) // BAD

// After
let myConstantSalt1 = Data("abcdef123456") // $ Source
let _ = myEncryptor.key(..., salt: myConstantSalt1, ...) // $ Alert

Lines with pre-existing // BAD, descriptive text comments were restructured to // $ Alert // BAD, descriptive text so the $ appears first in the comment content (required by the inline expectation parser).

Skipped tests (12)

Test	Reason
4× PrintAst.qlref	AST dump queries
library-tests/ast/Errors.qlref	No @kind metadata
AlertSuppression.qlref	@kind alert-suppression
Diagnostics/ExtractedFiles.qlref	@kind diagnostic
Diagnostics/SuccessfullyExtractedLines.qlref	@kind metric
CWE-757/InsecureTLS.qlref	file:// synthetic alert locations in #select (9 rows)
CWE-321/HardcodedEncryptionKey.qlref	file:// synthetic alert location
CWE-078/CommandInjection.qlref	file:// synthetic alert locations
CWE-135/StringLengthConflation.qlref	file:// synthetic source locations (13 rows)

Original prompt

Follow these instructions for Swift.
qlref conversion instructions

Here’s the repeatable recipe for converting a CodeQL language’s .qlref tests to inline expectation tests.

Overall Workflow

1. cd into the language folder.

2. Run grep -rL "InlineExpectationsTestQuery" --include="*.qlref".

3. Ignore generated .actual files, .testproj copies, and other generated artifacts.

4. For each .qlref, resolve the referenced query and read its metadata.

5. Classify each test before editing:

  - convert now for standard @kind problem and @kind path-problem

  - skip for anything else, with a short concrete reason

6. Inspect the existing .expected file before converting:

  - If it is empty and the query is a standard problem or path-problem query, usually convert by adding the postprocessor only. No source comments are needed.

  - If it is non-empty, add inline expectation comments to the source files at the expected result locations, then add the postprocessor to the .qlref.

7. Keep the existing .expected file. Do not edit generated expected files by hand, even when converting.

8. Edit only source files and .qlref files.

9. Maintain a skip list for tests not converted, with exact paths and short reasons.

10. Do not make a PR.

.qlref Formatting

For a single postprocessor, use the compact form:

query: path/to/Query.ql

postprocess: utils/test/InlineExpectationsTestQuery.ql

If the .qlref already has other postprocessors, use list form:

query: path/to/Query.ql

postprocess:

  - utils/test/PrettyPrintModels.ql

  - utils/test/InlineExpectationsTestQuery.ql

If the original .qlref was a plain single-line query reference, convert it to query: form only when adding postprocess:.

Always preserve existing postprocessors.

Expectation Comments

Use the language’s supported inline comment syntax from its inline expectation implementation. Typical tags are:

$ Alert

$ Source

$ Sink

For example, in Java or Kotlin:

sink(userInput); // $ Alert

For problem queries, add Alert at each expected result location.

For path-problem queries, add:

• Alert at the alert location.

• Source at the source location, unless the source is on the same line as the alert.

• Sink at the sink location when it is a distinct relevant location.

Multi-line result locations should be annotated on the end line.

If more than one result is expected on the same line for the same query, just use a single tag in the comment, for example:

foo(bar); // $ Alert

Only add comments to real source files that belong to the test. Do not annotate generated copies, shared vendored fixtures, or shared stubs unless that is clearly the right ownership boundary for the test.

Empty .expected Files

If a problem or path-problem test already has an empty .expected file, the default conversion is:

1. Leave .expected untouched.

2. Add postprocess: utils/test/InlineExpectationsTestQuery.ql to the .qlref.

3. Add no source comments unless the test actually has expected results that are currently represented elsewhere.

This is common in zero-result tests.

Query Ids

Do not add [query-id] by default.

Only qualify tags when needed, usually when more than one inline expectation test in the same effective test folder or database can see the same source comments. Then use the query’s metadata id:

response.write(x); // $ Alert[java/xss]

Apply this to Alert, Source, and Sink only for the comments that need disambiguation.

If you add query ids in a shared folder, validate the owner queries for those comments as well, not just the query you were editing.

Good Skip Reasons

Skip, or revert to non-inline, when conversion would make the test misleading, brittle, or noisy. Common reasons:

• The query is not @kind problem or @kind path-problem.

• The query is a telemetry, metrics, table, definitions, diagnostic, extractor-information, stub-generation, or tool-specific query.

• The test is an AST dump or similar structural output test, such as PrintAst.

• The selected result text is inside a string literal, raw string, heredoc, text block, XML literal, or multiline literal and adding a comment changes the selected text.

• The expected location is inside documentation syntax where adding a comment changes what the query reports, such as Javadoc tags.

• The .expected output is nonstandard enough that mechanical conversion is risky.

• Results are in shared stubs, generated copies, vendored files, or synthetic/test-project paths where inline comments would pollute shared fixtures.

• The alert, source, or sink locations are too dense or ambiguous to make the source understandable.

• The language’s inline expectation implementation does not support the source file or comment style needed.

• The query uses a special result kind or framework behavior, such as alert-...

Created from VS Code.

[owen-mc]

@copilot In swift/ql/test/query-tests/Security/CWE-020/ you have converted two tests to inline expectations tests, but you have used // $ Alert for all the alerts. This means that each test will expect alerts at places that the other test has a result. You have to use the optional query id form in this situation: // $ Alert[swift/missing-regexp-anchor]. Please fix this issue in this folder and the others where it applies: swift/ql/test/query-tests/Security/CWE-311/ and swift/ql/test/query-tests/Security/CWE-328/. You can use the .expected files for each test to see which alerts belong to which test. (If there is an alert for >1 test on one line then add comments with the query id for all relevant tests.) If you have a codeql binary available then you may run it to see the test output.

[Copilot]

@copilot In swift/ql/test/query-tests/Security/CWE-020/ you have converted two tests to inline expectations tests, but you have used // $ Alert for all the alerts. This means that each test will e...

Fixed in the latest commit. For each of the three folders:

• CWE-020: test.swift now uses // $ Alert[swift/incomplete-hostname-regexp] or // $ Alert[swift/missing-regexp-anchor] per-line based on which query produces that result. Lines 81 and 84 (where both queries produce one result each) keep bare // $ Alert. SemiAnchoredRegex.swift and UnanchoredUrlRegex.swift now use [swift/missing-regexp-anchor] throughout (only MRA produces results there). I also fixed lines with multiple results that previously had only one Alert annotation (e.g., line 65 needs two, line 77 needs three IHR + one MRA).

• CWE-311: Since the files are disjoint (CleartextStorageDatabase covers SQLite/CoreData/GRDB/Realm; CleartextTransmission covers Alamofire/Send/URL), all annotations now use the appropriate [swift/cleartext-storage-database] or [swift/cleartext-transmission] qualifier.

• CWE-328: testCryptoKit.swift and testCryptoSwift.swift now use [swift/weak-password-hashing] for password variables (passwd/passwdArray/etc.) and [swift/weak-sensitive-data-hashing] for sensitive data variables (cert/account_no/credit_card_no/phoneNumberArray/etc.), matching the split in the expected files.

[Copilot]

Pull request overview

This PR migrates Swift security query tests from .expected-file-driven assertions to inline expectation markers (// $ Alert, // $ Source, // $ Sink) in the Swift test sources, using utils/test/InlineExpectationsTestQuery.ql as a postprocessor to validate results.

Changes:

• Updated multiple Swift security .qlref tests to add postprocess: utils/test/InlineExpectationsTestQuery.ql.
• Added inline expectation annotations across Swift test sources to represent expected alerts/sources/sinks directly in code.
• Tweaked Swift’s inline-expectations comment extraction to trim comment contents.

Show a summary per file

File	Description
swift/ql/test/query-tests/Security/CWE-916/test.swift	Adds inline $ Source/$ Alert markers for insufficient hash-iteration test cases.
swift/ql/test/query-tests/Security/CWE-916/InsufficientHashIterations.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-760/test.swift	Adds inline markers for constant-salt test cases.
swift/ql/test/query-tests/Security/CWE-760/rncryptor.swift	Adds inline markers for constant-salt test cases and tracks known missing alerts.
swift/ql/test/query-tests/Security/CWE-760/ConstantSalt.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-730/tests.swift	Adds inline markers for regex-injection alerts/sources and known spurious results.
swift/ql/test/query-tests/Security/CWE-730/RegexInjection.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-730/RegexInjection.expected	Regenerated/rewritten expected output layout (now includes #select at top).
swift/ql/test/query-tests/Security/CWE-328/WeakSensitiveDataHashing.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-328/WeakPasswordHashing.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-328/testCryptoSwift.swift	Adds query-id-qualified inline $ Alert[...] markers to disambiguate overlapping weak-hash queries.
swift/ql/test/query-tests/Security/CWE-328/testCryptoKit.swift	Adds query-id-qualified inline $ Alert[...] and $ Source[...] markers; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-327/test.swift	Adds inline $ Source/$ Alert markers for ECB encryption cases.
swift/ql/test/query-tests/Security/CWE-327/ECBEncryption.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-312/testUserDefaults.swift	Adds query-id-qualified inline markers for cleartext storage (preferences) cases.
swift/ql/test/query-tests/Security/CWE-312/testNSUbiquitousKeyValueStore.swift	Adds query-id-qualified inline markers for iCloud key-value storage cases.
swift/ql/test/query-tests/Security/CWE-312/CleartextStoragePreferences.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-312/CleartextStoragePreferences.expected	Regenerated/rewritten expected output layout (now includes #select at top).
swift/ql/test/query-tests/Security/CWE-312/cleartextLoggingTest.swift	Adds query-id-qualified inline markers for cleartext logging cases; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-311/testURL.swift	Adds query-id-qualified inline markers for cleartext transmission in URL construction; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-311/testSwiftUI.swift	Adds a query-id-qualified inline missing-alert marker in a SwiftUI-specific case.
swift/ql/test/query-tests/Security/CWE-311/testSend.swift	Adds query-id-qualified inline markers for cleartext transmission via NWConnection.send.
swift/ql/test/query-tests/Security/CWE-311/testRealm2.swift	Adds query-id-qualified inline markers for cleartext database storage via Realm; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-311/testRealm.swift	Adds query-id-qualified inline markers for cleartext database storage via Realm.
swift/ql/test/query-tests/Security/CWE-311/testGRDB.swift	Adds query-id-qualified inline markers for cleartext database storage via GRDB APIs.
swift/ql/test/query-tests/Security/CWE-311/testCoreData2.swift	Adds query-id-qualified inline markers for CoreData persistence cases; includes spurious markers.
swift/ql/test/query-tests/Security/CWE-311/testCoreData.swift	Adds query-id-qualified inline markers for CoreData persistence cases; includes missing/spurious markers.
swift/ql/test/query-tests/Security/CWE-311/testAlamofire.swift	Adds query-id-qualified inline markers for cleartext transmission via Alamofire; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-311/sqlite3_c_api.swift	Adds query-id-qualified inline markers for cleartext database storage via sqlite3 C APIs.
swift/ql/test/query-tests/Security/CWE-311/SQLite.swift	Adds query-id-qualified inline markers for cleartext database storage via SQLite.swift APIs.
swift/ql/test/query-tests/Security/CWE-311/CleartextTransmission.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-311/CleartextStorageDatabase.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-259/test.swift	Adds inline markers for constant-password test cases.
swift/ql/test/query-tests/Security/CWE-259/rncryptor.swift	Adds inline $ Source/$ Alert markers for constant-password flows.
swift/ql/test/query-tests/Security/CWE-259/ConstantPassword.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-134/UncontrolledFormatString.swift	Adds inline $ Source/$ Alert markers for uncontrolled format-string cases; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-134/UncontrolledFormatString.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-1333/ReDoS.swift	Adds inline $ Alert markers for ReDoS cases, preserving explanatory comments.
swift/ql/test/query-tests/Security/CWE-1333/ReDoS.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-1204/test.swift	Adds inline $ Source/$ Alert markers for static IV usage.
swift/ql/test/query-tests/Security/CWE-1204/StaticInitializationVector.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-1204/rncryptor.swift	Adds inline $ Source/$ Alert markers for static IV usage in RNEncryptor flows.
swift/ql/test/query-tests/Security/CWE-116/test.swift	Adds inline $ Alert markers for bad tag-filter regex variants.
swift/ql/test/query-tests/Security/CWE-116/BadTagFilter.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-089/sqlite3_c_api.swift	Adds inline $ Source/$ Alert markers for SQL injection via sqlite3 APIs.
swift/ql/test/query-tests/Security/CWE-089/SQLite.swift	Adds inline $ Source/$ Alert markers for SQL injection via SQLite.swift APIs.
swift/ql/test/query-tests/Security/CWE-089/SqlInjection.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-089/other.swift	Adds inline $ Source/$ Alert markers for heuristic SQL injection stubs; retains known missing alerts.
swift/ql/test/query-tests/Security/CWE-089/GRDB.swift	Adds inline $ Source/$ Alert markers for SQL injection via GRDB APIs.
swift/ql/test/query-tests/Security/CWE-079/UnsafeWebViewFetch.swift	Adds inline $ Source/$ Alert markers for unsafe WebView fetch flows.
swift/ql/test/query-tests/Security/CWE-079/UnsafeWebViewFetch.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-079/UnsafeWebViewFetch.expected	Regenerated/rewritten expected output layout (now includes #select at top).
swift/ql/test/query-tests/Security/CWE-022/UnsafeUnpack/UnsafeUnpack.swift	Adds inline $ Source/$ Alert markers for unsafe unzip/unpack flows.
swift/ql/test/query-tests/Security/CWE-022/UnsafeUnpack/UnsafeUnpack.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-020/UnanchoredUrlRegex.swift	Adds query-id-qualified inline markers for missing-regexp-anchor cases, including missing/spurious annotations.
swift/ql/test/query-tests/Security/CWE-020/test.swift	Adds query-id-qualified inline markers for incomplete-hostname/missing-anchor regex cases.
swift/ql/test/query-tests/Security/CWE-020/SemiAnchoredRegex.swift	Adds query-id-qualified inline markers for semi-anchored regex cases.
swift/ql/test/query-tests/Security/CWE-020/MissingRegexAnchor.qlref	Adds inline-expectations postprocessor.
swift/ql/test/query-tests/Security/CWE-020/IncompleteHostnameRegex.qlref	Adds inline-expectations postprocessor.
swift/ql/lib/utils/test/internal/InlineExpectationsTestImpl.qll	Adjusts Swift comment content extraction used by the inline expectations framework.

Copilot's findings

• Files reviewed: 62/62 changed files
• Comments generated: 3

[jketema]

In cases like these, where we just had // BAD [NOT DETECTED] with no additional text, should we just drop the // BAD [NOT DETECTED]?

[owen-mc]

My policy has been to just keep all the original comments, as there are too many to make decisions on all of them. You are very welcome to either push a commit to this PR clearing them up or do it as a follow-up.

[jketema]

Should this be marked SPURIOUS

[owen-mc]

It isn't totally clear to me from the comment. I added SPURIOUS and MISSING as a best-effort based on searching for comments like "not detected", and I don't consider it a blocker to this PR to get those all correct. Gopefully someone with more language knowledge can interpret what this comment means.

[jketema]

Looking at 2622de9, not having a SPURIOUS here is seems the right thing.

[jketema]

LGTM