Skip to content

[RNE Rewrite] feat: add keypoint detection task pipeline#1280

Merged
barhanc merged 14 commits into
rne-rewritefrom
@bh/keypoint-detection
Jun 25, 2026
Merged

[RNE Rewrite] feat: add keypoint detection task pipeline#1280
barhanc merged 14 commits into
rne-rewritefrom
@bh/keypoint-detection

Conversation

@barhanc

@barhanc barhanc commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Adds kepoint detection task pipeline implementation and corresponding example screen in computer-vision app.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

Screenshots

Related issues

Closes #1241

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@barhanc barhanc self-assigned this Jun 24, 2026
@barhanc barhanc added refactoring feature PRs that implement a new feature labels Jun 24, 2026
@barhanc barhanc linked an issue Jun 24, 2026 that may be closed by this pull request
[RNE Rewrite] CV - add keypoint detection (including pose estimation and BlazeFace) pipeline #1241
Closed
@barhanc barhanc force-pushed the @bh/keypoint-detection branch from 3eb2f56 to 33e6160 Compare June 24, 2026 12:20
@barhanc

barhanc commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@msluszniak Where should I put the blazeface model, we already have the v0.10.0 tag on the HF, but to work with the new pipeline the model was exported in slightly different manner (export script on the internal gitlab)? cc: @chmjkb

@barhanc barhanc marked this pull request as ready for review June 24, 2026 14:31
@barhanc barhanc requested a review from msluszniak June 24, 2026 14:31
@msluszniak

msluszniak commented Jun 24, 2026

Copy link
Copy Markdown
Member

Blazeface was never released officially. So I would personally force push this on v0.10.0 tag on Hugging Face.

msluszniak
msluszniak reviewed Jun 24, 2026
View reviewed changes
Comment thread packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp
Comment thread packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp Outdated
Comment thread packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp Outdated
barhanc and others added 2 commits June 24, 2026 16:51
@barhanc
refactor(box_ops): remove else after return in parse functions
b9ba922
@barhanc @msluszniak
Apply suggestion from @msluszniak
d92c95f 
Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>
msluszniak
msluszniak reviewed Jun 24, 2026
View reviewed changes
Comment thread packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp Outdated
Comment thread apps/computer-vision/app/keypoint/index.tsx Outdated
Comment thread packages/react-native-executorch/src/extensions/cv/tasks/keypointDetection.ts
@barhanc barhanc requested a review from msluszniak June 24, 2026 18:44
@msluszniak

msluszniak commented Jun 24, 2026

Copy link
Copy Markdown
Member

Check updated HuggingFace repos:

Could you add which repos are updated?

@barhanc

barhanc commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Fixed. Sorry, I was going to add links after re-exporting models but forgot. Also, I've added the YOLOv8n pose estimation model. When implementing PoC I was sure it was the same as YOLO26, but turns out these are different models, so now we have both. I can remove the YOLOv8n if you want, since YOLO26 is like a better version.

@msluszniak

msluszniak commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Regarding HF, blazeface is missing content of README.md and just to make sure: Are the reexported models indeed compatible with ET 1.2 or it's typo and it's compatible with 1.3 (currently supported on main)? Also yolo26 has config in xnnpack dir, yolov8 doesn't, same for blazeface. Regarding yolov8, if it's fastee than we can keep it. If not, probably worth removing.

@barhanc

barhanc commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Are the reexported models indeed compatible with ET 1.2 or it's typo and it's compatible with 1.3 (currently supported on main)?

They were exported using 1.2.0. Also currently the binaries we copy from PoC are also 1.2.0.

Regarding yolov8, if it's fastee than we can keep it. If not, probably worth removing.

Ok, then I will remove it.

Also yolo26 has config in xnnpack dir, yolov8 doesn't, same for blazeface.

Should I add the configs to the ones that don't have it or remove it from the ones that have? Where are these configs used actually?

@msluszniak

msluszniak commented Jun 24, 2026

Copy link
Copy Markdown
Member

We should rather add these configs. @NorbertKlockiewicz added recently configs for other models. They help in re-exports, debbugability, are readable for agents. Probably there are some more which Norbert can name.

msluszniak
msluszniak reviewed Jun 25, 2026
View reviewed changes
Comment thread packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp Outdated
@barhanc @msluszniak
Update packages/react-native-executorch/cpp/extensions/cv/box_ops.cpp
cfc10f4 
Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>
@barhanc barhanc merged commit bf00556 into rne-rewrite Jun 25, 2026
2 checks passed
@barhanc barhanc deleted the @bh/keypoint-detection branch June 25, 2026 11:01
@barhanc barhanc mentioned this pull request Jun 25, 2026
Closed
msluszniak added a commit that referenced this pull request Jun 25, 2026
@msluszniak
[RNE Rewrite] feat: add keypointDetection feature mapping (#1280)
ab793e1 
Rebased onto rne-rewrite incl. #1280 keypoint detection. Rename the
poseEstimation FEATURE_MAP entry to keypointDetection (tracks the
useKeypointDetector hook) and set backends to xnnpack+coreml+mlx (RF-DETR
keypoint ships CoreML + MLX variants) + opencv. Declare it in the
computer-vision demo app.
msluszniak added a commit that referenced this pull request Jun 25, 2026
@msluszniak
[RNE Rewrite] feat: add keypointDetection feature mapping (#1280)
fd3ac2e 
Rebased onto rne-rewrite incl. #1280 keypoint detection. Rename the
poseEstimation FEATURE_MAP entry to keypointDetection (tracks the
useKeypointDetector hook) and set backends to xnnpack+coreml+mlx (RF-DETR
keypoint ships CoreML + MLX variants) + opencv. Declare it in the
computer-vision demo app.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@msluszniak msluszniak msluszniak approved these changes

Assignees

@barhanc barhanc

Labels

feature PRs that implement a new feature refactoring

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[RNE Rewrite] CV - add keypoint detection (including pose estimation and BlazeFace) pipeline

2 participants

@barhanc @msluszniak