feat: add RavenForCausalLM (Huginn) depth-recurrent adapter#1520
Merged
Conversation
Contributor
Author
|
@jlarson4 ready for review. Rebased against dev — conflicts resolved. |
Adds a TransformerBridge architecture adapter for RavenForCausalLM (tomg-group-umd/huginn-0125), a depth-recurrent decoder with a prelude / weight-tied recurrent core / coda structure, resolving TransformerLensOrg#1469. - raven.py delegates the recurrence to the remote-code HF forward and maps the three physical block lists (prelude / core_block / coda) via SSMBlockBridge, with combined-QKV native attention and a gated MLP. Sets applicable_phases=[] (a random initial latent state and post-residual sandwich norms diverge from the verify_models phases) and supports_fold_ln=False (ln_f is reused mid-network). prepare_loading patches Huginn's remote code for transformers v5 (tied-weights-keys dict form + a weight re-init guard). - Registers in the adapter factory, the model registry (canonical author and description) and supported_models.json; surfaces the recurrence-shape config via both _HF_PASSTHROUGH_ATTRS lists. - Adds synthetic-config unit tests and CI-gated integration tests.
mukund1985
force-pushed
the
feat/raven-adapter
branch
from
July 17, 2026 05:41
5653cbc to
9336e87
Compare
1 task
Collaborator
|
@mukund1985 You've got some CI failures to resolve, please take a look at those! I'll have a more thorough review soon, but you can start there |
jlarson4
reviewed
Jul 20, 2026
Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
…MBlockBridge Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
Contributor
Author
|
Addressed in the latest commits - extracted |
Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
Raven uses a combined Wqkv projection (no separate q/k/v submodules), so AttentionBridge's default hook_q/hook_k/hook_v aliases (which target q.hook_out / k.hook_out / v.hook_out) are unresolvable. The upstream test_every_hook_alias_resolves_to_hookpoint audit catches these as 9 dead aliases across prelude / core_block / coda. Strip them by setting an instance-level hook_aliases that omits those three keys, leaving only the aliases that have real HookPoint targets. Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
… entry Brings in the RWKV-7 additions from TransformerLensOrg#1521 that landed in dev: - _bridge_builder.py: added RWKV-7 passthrough attrs (num_heads, value_dim, decay/gate/a/v_low_rank_dim, norm_first, norm_bias, fuse_norm, attn_mode, hidden_act) - generate_report.py: added RWKV7ForCausalLM description entry Signed-off-by: Mukund Pandey <mukund.pandey@gmail.com>
Collaborator
|
This one looks good as well! Once the CI completes I will merge, thanks @mukund1985! |
Contributor
Author
|
@jlarson4 all CI checks have passed. Ready to merge when you are. |
38 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a TransformerBridge adapter for
RavenForCausalLM(tomg-group-umd/huginn-0125), a depth-recurrent transformer that performs latent reasoning by iterating a weight-tied core block a variable number of times at inference.Closes #1469.
Architecture
Huginn has a three-phase structure under
model.transformer:num_stepstimes at runtime (defaultmean_recurrence=32). Each step re-injects the prelude output via a learnedadapterlinearSandwichBlock uses a post-residual norm pattern (
x = norm_2(attn(norm_1(x)) + x)).BlockBridgeenforces a pre-norm hook flow that does not exist here, soSSMBlockBridgeis used for all three block lists to delegate the full forward unchanged.Key implementation notes
transformers v5 compatibility. The remote code targets transformers 4.44 where
_tied_weights_keysis a list. Under v5,tie_weights()expects a dict. Fixed inprepare_loading()by rewriting it to{"lm_head.weight": "transformer.wte.weight"}, plus the standard_init_weightsre-randomization guard.Non-deterministic forward.
iterate_forwardseeds the recurrence withtorch.randn_like. Integration tests pin the RNG identically before bridge and HF calls.Hook exposure.
core_block.{i}.hook_in/hook_outfire once per recurrence step, enabling cross-step logit lens, fixed-point analysis, and cross-step activation patching.Registration
All four required sites updated per AGENTS.md:
supported_architectures/__init__.py,factories/architecture_adapter_factory.py(key:RavenForCausalLM),tools/model_registry/__init__.py,tools/model_registry/generate_report.py,tools/model_registry/data/supported_models.json.Tests
test_raven_adapter.py): 58 tests, synthetic config, no weight loadingtest_raven_adapter.py): 16/16 pass locally against the full fp32 checkpoint, including forward logit parity vs HF eager, hook shapes across all three phases, and recurrent-core hook firing count. CI-gated on network/memory budget.TestRegistrySyncedWithFactory(4 invariants): all pass