feat(#410): retrain mesh segmentation model — 31.5% → 94.7% on rig-truth eval#787
Conversation
|
Warning Review limit reached
Next review available in: 29 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThis PR updates the mesh segmentation model documentation and rewrites the training/export pipeline in ChangesMesh Segmentation v2 Update
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ebc54bea2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| caps_dir = _unit_dirs(nc, rng) | ||
| which = rng.random(nc) < 0.5 | ||
| caps = np.where(which[:, None], p0 + caps_dir * r, p1 + caps_dir * r) |
There was a problem hiding this comment.
During v2 training, every limb/neck/tail that goes through capsule_surf() samples cap directions from a full sphere and attaches the whole sphere to each endpoint. Half of those cap points land on the inward side of the endpoint sphere, inside the cylinder rather than on the exterior capsule surface, so the supposedly surface-only training set still injects non-mesh-surface points for all capsule parts; restrict each cap to its outward hemisphere to match real mesh vertices.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 62bb418 — cap directions are now reflected into their outward hemispheres, so every sampled point lies exactly on the capsule surface (verified: all points at distance r from the axis segment). Note: the v2 weights currently on HF were trained before this purity fix; the caps are a small fraction of each limb's points, and the model is validated at 94.7% rig-truth accuracy, so the fix lands in the data pipeline for the next retrain rather than triggering an immediate re-release.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
scripts/export-meshseg-onnx.py (1)
472-479: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate file-collection logic between
load_real_data()andcheck_real().Both functions independently glob/collect files from
paths:files = [] for pth in paths: files += sorted(glob.glob(os.path.join(pth, "*.json"))) if os.path.isdir(pth) else [pth]Extract a shared helper to avoid drift between the two.
♻️ Proposed refactor
+def _collect_json_files(paths): + files = [] + for pth in paths: + files += sorted(glob.glob(os.path.join(pth, "*.json"))) if os.path.isdir(pth) else [pth] + return files + + def load_real_data(paths, aug, seed, exclude=(), only=()): rng = np.random.default_rng(seed + 777) - files = [] - for pth in paths: - files += sorted(glob.glob(os.path.join(pth, "*.json"))) if os.path.isdir(pth) else [pth] + files = _collect_json_files(paths) files = [f for f in files if not any(e in os.path.basename(f) for e in exclude)]def check_real(paths): """--check-real: print canonicalisation sanity for every mined file.""" - files = [] - for pth in paths: - files += sorted(glob.glob(os.path.join(pth, "*.json"))) if os.path.isdir(pth) else [pth] + files = _collect_json_files(paths)Also applies to: 506-511
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/export-meshseg-onnx.py` around lines 472 - 479, The file-collection logic in load_real_data() and check_real() is duplicated and should be centralized to prevent drift. Extract the shared path-to-json gathering loop into a helper that both functions call, keeping the existing filtering behavior in load_real_data() intact. Use the unique symbols load_real_data() and check_real() to locate and replace the repeated globbing block with the shared helper.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/export-meshseg-onnx.py`:
- Around line 380-470: The current side sanity check in check_real() is
self-referential because canonicalise() already recomputes Lc from Pc, so a
global L/R swap can still pass as ok. Update check_real() to use an independent
reference-based left/right validation against the original labels or raw
geometry before canonicalisation, using canonicalise(), Pc, and Lc; if no such
reference exists, remove the arm/leg side assertion and keep only the geometry
consistency checks.
In `@scripts/fetch-training-rigs.sh`:
- Around line 28-32: The Quaternius mirror note in the sources list is too broad
and should be narrowed to the exact CC0 subfolders actually used. Update the
entry referenced by the comment in fetch-training-rigs.sh so it mentions only
the specific “FreeModels by Quaternius[Patreon]/Characters and Animals/” paths
used by the corpus, and avoid labeling the entire GitHub mirror as CC0. Keep the
exclusion of the “[Patreon Exclusive]” folders explicit so the recorded source
scope matches the data selection.
---
Nitpick comments:
In `@scripts/export-meshseg-onnx.py`:
- Around line 472-479: The file-collection logic in load_real_data() and
check_real() is duplicated and should be centralized to prevent drift. Extract
the shared path-to-json gathering loop into a helper that both functions call,
keeping the existing filtering behavior in load_real_data() intact. Use the
unique symbols load_real_data() and check_real() to locate and replace the
repeated globbing block with the shared helper.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a7bc9a20-37d3-46bd-8449-b42e68126d27
📒 Files selected for processing (6)
.gitignoreCLAUDE.mdTHIRD_PARTY_AI_MODELS.mddocs/MESH_SEGMENTATION_STRATEGY.mdscripts/export-meshseg-onnx.pyscripts/fetch-training-rigs.sh
…uth eval The v1 meshseg.onnx scored 31.5% per-vertex accuracy against exact rig-derived ground truth (near-random limbs; chibi characters collapsed to all-torso). Root causes: volumetric instead of surface point sampling, no chibi proportions, uniform per-part vertex density, LEFT/RIGHT flipped vs the rig-prior convention, and a 1024-vs-4096 train/inference point-count mismatch. export-meshseg-onnx.py v2 (same CLI, new pipeline): - surface-sampled connected synthetic bodies in three plans (humanoid incl. chibi/lanky, quadruped with side-labelled legs, biped-with-tail), randomised per-part density, feet/muzzle +Z facing cues, correct handedness (LEFT at +X) - mined CC0 Quaternius rigs (ledger in the corpus SOURCES.md): the loader canonicalises arbitrarily-oriented bind-pose clouds from their own labels (scored up-axis candidates, left from limb axis, 180° yaw fix) and geometrically reassigns arm/leg sides (miner side detection was up to ~30% wrong on some rigs); incoherent files are excluded - deeper PointNet++-style net (two kNN blocks, ~1 MB), unknown-masked class-weighted loss, phase-2 fine-tune at the app's 4096-point inference size - --check-real mode to audit canonicalisation of a mined corpus Results (exact rig-truth eval on out-of-distribution characters kept out of training): 94.7% overall vs 31.5% baseline; held-out CC0 rigs 97.0%. End-to-end via qtmesh segment: rigged meshes reproduce rig-truth counts; unrigged OBJ path correctly lateralises all limbs. Model uploaded to the HF models repo (segment/meshseg.onnx). Docs: docs/MESH_SEGMENTATION_STRATEGY.md (failure analysis, results, multi-category roadmap for vegetation/vehicle/building segmentation), THIRD_PARTY_AI_MODELS.md + CLAUDE.md refreshed, fetch-training-rigs.sh records the stable CC0 mirror source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fernandotonon
force-pushed
the
feat/meshseg-v2-410
branch
from
3ebc54b to
09b06eb
Compare
…ped mirror provenance - capsule_surf: reflect cap directions into their outward hemispheres so every sampled point lies ON the capsule surface (half the cap points previously landed inside the cylinder — off-surface noise in the supposedly surface-only training set). Verified: all sampled points now at exactly distance r from the axis segment. - check_real: the post-canonicalisation L/R centroid test was self-referential (canonicalise() rewrites sides geometrically, so it held by construction). The pass criterion is now geometry-only, and the independent miner-quality signal is the new `sidefix` fraction — how many limb points' MINED side labels disagreed with the geometric side (a whole-file swap surfaces as sidefix ≈ 100%). - fetch-training-rigs.sh: scope the mirror provenance note to the exact CC0 Quaternius subfolders and require per-pack SOURCES.md rows instead of a blanket [CC0 content] label for the whole mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
1 participant
Summary
The shipped
meshseg.onnx(v1) segmented real meshes poorly — 31.5% per-vertex accuracy against exact rig-derived ground truth, with near-random limb labels and chibi-proportioned characters collapsing to all-torso. This PR retrains it end to end; the new model scores 94.7% on the same eval and is already uploaded to the HF models repo (segment/meshseg.onnx), so existing installs pick it up on next first-use download. No C++ changes — the ONNX contract is unchanged.Root causes of the v1 failure (all confirmed empirically)
MeshSegmenter::predict()samples 4096What v2 does (
scripts/export-meshseg-onnx.py, same CLI)partForBoneName), biped-with-tail — with randomised per-part density and correct handednessSOURCES.md): the loader canonicalises arbitrarily-oriented bind-pose clouds from their own labels (scored up-axis candidates, limb-axis left, 180° yaw fix for backward-facing animals) and geometrically reassigns arm/leg sides — the miner's bone-name side detection was up to ~30% wrong on some rigs; incoherent files are excluded--check-realmode audits canonicalisation of a mined corpusResults (exact rig-derived ground truth)
End-to-end via
qtmesh segment(3.14.0-dev ONNX build): rigged meshes reproduce rig-truth part counts almost exactly; the same character exported to unrigged OBJ (pure model path) correctly lateralises all limbs with only shoulder/hip boundary bleed. Verified visually in the GUI.Docs
docs/MESH_SEGMENTATION_STRATEGY.md— failure analysis, results, and the proposed multi-category roadmap (vegetation/vehicle/building label sets via per-category models + name-mined CC0 packs)THIRD_PARTY_AI_MODELS.md/CLAUDE.mdrefreshed;fetch-training-rigs.shrecords the stable CC0 mirror source🤖 Generated with Claude Code