Skip to content

Improve comparison base resolution for review skills - #27827

Merged
Craig Macomber (Microsoft) (CraigMacomber) merged 5 commits into
microsoft:mainfrom
CraigMacomber:improve-comparison-base-skill
Aug 10, 2026
Merged

Craig Macomber (Microsoft) (CraigMacomber) merged 5 commits into
microsoft:mainfrom
CraigMacomber:improve-comparison-base-skill

Conversation

@CraigMacomber

@CraigMacomber Craig Macomber (Microsoft) (CraigMacomber) commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Description

The existing review skill was often counterproductive for me as it was very specific about what to review against, often worse than what the model would come up with on its own. These changes correct its assumptions, making the skill much more generally useful, even with less powerful models, explicitly directing it to handle removes with different names (like my upstream), compare to the merge point with the target branch not its head, handle local changes (untracked, modifications and staged changes) etc.

The new approach checks for an existing PRs as a heuristic to find the correct upstream branch and remote, then looks for an upstream for the https://github.com/microsoft/FluidFramework repo, regardless of its name.

It also warns if the merge base and target branch are too out of sync by commit count, not just diff size (which was slow and confusing when I hit it in my fork due to using the wrong remote).

I also refactored our two existing skills which hard coded origin/main to both use this extracted common logic for improved consistency and maintainability.

I used this skill to review itself, with a couple models and at a couple points in the process (like when it was just local changes on main, when it was staged on a branch, and after the PR was created. Even Haiku 4.5 could follow the steps and produce a review using this skill (its review quality wasn't great, but it found the correct diff to review).

Reviewer Guidance

The review process is outlined on this wiki page.

@github-actions github-actions Bot added area: tools area: repo Repo related work area: website base: main PRs targeted against main branch labels Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Hi! Thank you for opening this PR. Want me to review it?

Based on the diff (187 lines, 3 files), I've queued these reviewers:

  • Correctness — logic errors, race conditions, lifecycle issues
  • Security — vulnerabilities, secret exposure, injection
  • API Compatibility — breaking changes, release tags, type design
  • Performance — algorithmic regressions, memory leaks
  • Testing — coverage gaps, hollow tests

How this works

  • Adjust the reviewer set by ticking/unticking boxes above. Reviewer toggles alone don't trigger anything.

  • Tick Start review below to dispatch the review fleet.

  • After review finishes, tick Start review again to request another run — it auto-resets after each dispatch.

  • This comment updates as new commits land; your reviewer selections are preserved.

  • Start review

@CraigMacomber
Craig Macomber (Microsoft) (CraigMacomber) marked this pull request as ready for review August 3, 2026 21:29
Copilot AI review requested due to automatic review settings August 3, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the “review” and “api-changes” skills to stop assuming origin/main and instead resolve a more accurate comparison base (PR base branch when available, otherwise the canonical microsoft/FluidFramework upstream). It adds a reusable “comparison-base” skill to centralize that logic and improve behavior across forks, renamed remotes, and local-only change scenarios.

Changes:

  • Added a new .claude/skills/comparison-base skill to resolve PR target/upstream and select a comparison commit on the target’s first-parent history, with divergence detection.
  • Updated the review skill to use the resolved base commit and consistently choose local-vs-remote diff endpoints (including staged/unstaged/untracked in local reviews).
  • Updated the api-changes skill to compare API report diffs against the resolved base commit (not origin/main).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.claude/skills/review/SKILL.md Switches the review workflow to use resolved target/base commit and supports local vs remote diffing consistently.
.claude/skills/comparison-base/SKILL.md New reusable workflow that resolves target branch + selects a stable comparison commit and checks divergence.
.claude/skills/api-changes/SKILL.md Updates API-change detection to diff against the resolved comparison base (and working tree).
Suppressed comments (1)

.claude/skills/comparison-base/SKILL.md:110

  • Since the skill now resolves $TARGET_REPOSITORY (owner/repo), it should be returned in the Outputs list so callers (e.g. review) can print an unambiguous repo:branch base in reports.
- `$TARGET_SOURCE`, `$TARGET_BRANCH`, and `$TARGET_REF`

Comment thread .claude/skills/api-changes/SKILL.md Outdated
Comment thread .claude/skills/comparison-base/SKILL.md Outdated
Comment thread .claude/skills/comparison-base/SKILL.md Outdated
Comment on lines +34 to +38
First determine whether an open PR exists for the review branch. Prefer active PR metadata from the editor's PR integration when available, but verify its head repository and branch match `$REVIEW_REPOSITORY` and `$REVIEW_HEAD_BRANCH` when those values are known. Its published head must also match `$REVIEW_OID` or be an ancestor of a locally-ahead `$REVIEW_REF`.

Otherwise, query the GitHub commit-to-PR API (`repos/{owner}/{repo}/commits/$REVIEW_OID/pulls`) for each GitHub repository named by configured remotes, including `microsoft/FluidFramework`. Treat HTTP 404 or 422 as no commit candidates and continue; locally-ahead or unpublished commits may not exist in the queried repository. If `$REVIEW_REPOSITORY` and `$REVIEW_HEAD_BRANCH` are known, also query open PRs by that full head repository owner plus branch so locally-ahead commits can resolve their published PR. Read `state`, `base.repo.clone_url`, `base.ref`, `head.repo.full_name`, `head.ref`, `head.sha`, and `html_url` from the REST response. Only accept open PRs whose head repository and branch match the known review identity and whose head SHA matches `$REVIEW_OID` or is its ancestor.

Do not rely on bare `gh pr view` or an unqualified branch name. These are ambiguous in fork checkouts. If multiple open PRs match, ask the user which PR to use.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know gh won't work in all cases, but I think these gh commands will reveal the right info to the agent in many cases and may be easier for it to follow:

gh pr view --json url,state,baseRefName,headRefName,headRefOid,headRepository
gh repo view --json nameWithOwner,defaultBranchRef

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have incorporated this approach as an option (with some tweaks to make it robust with no default repository), and also adjusted the process so it stop when it find a good branch to reduce extra tool calls.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the bundle size test has to do something similar where it figures out the base commit - but I guess it's easier in CI where the remotes are fixed.

@CraigMacomber

Copy link
Copy Markdown
Contributor Author

It seems like the bundle size test has to do something similar where it figures out the base commit - but I guess it's easier in CI where the remotes are fixed.

It's not just the fixed remotes, but in our PR CI you know that there is always a PR with a clear merge target. This skill can work without a PR, and even without a branch (review local changes). It only uses PRs as an optional heuristic mostly so that if your branch is merging into something other than main it can still give a good review.

@github-actions

Copy link
Copy Markdown
Contributor

🔗 Found some broken links! 💔

Run a link check locally to find them. See
https://github.com/microsoft/FluidFramework/wiki/Checking-for-broken-links-in-the-documentation for more information.

linkcheck output

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> fluid-framework-website@0.0.0 serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/
[ELIFECYCLE] Command failed with exit code 1.

@zhangxin511
zhangxin511 self-requested a review August 10, 2026 20:21
@CraigMacomber
Craig Macomber (Microsoft) (CraigMacomber) merged commit 63630aa into microsoft:main Aug 10, 2026
30 checks passed
@CraigMacomber
Craig Macomber (Microsoft) (CraigMacomber) deleted the improve-comparison-base-skill branch August 10, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: repo Repo related work area: tools area: website base: main PRs targeted against main branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants