Skip to content

add some important features from ruby > 3.0 and solve some issue - #282

Open
Magikdidi24 wants to merge 73 commits into
tree-sitter:masterfrom
Magikdidi24:master
Open

Magikdidi24 wants to merge 73 commits into
tree-sitter:masterfrom
Magikdidi24:master

Conversation

@Magikdidi24

Copy link
Copy Markdown
  • shadows variables
  • for range with begin only
  • functions ruby >3.0
  • bad return

@Magikdidi24
Magikdidi24 marked this pull request as ready for review October 20, 2025 12:31
a = 42
-> {
# This is also another a.
use a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate why this not the same as a on line 2771?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a mistake — it’s the same. I just used my own test case in my project and forgot to remove the comment.

@Magikdidi24
Magikdidi24 force-pushed the master branch 2 times, most recently from 0738b52 to 65bb643 Compare October 21, 2025 14:24
Committed src/ was out of sync with grammar.js (stale conflict entry).

Before (src/grammar.json conflicts):
  [["_expression","_chained_command_call"], ..., ["_statement","call"]]
After:
  [["_expression","_chained_command_call"], ...]
Expected tree listed a (comment) node that is not in the source.

Before:
      body: (block
        (comment)
        body: (block_body
After:
      body: (block
        body: (block_body
…e-sitter#290, tree-sitter#244, tree-sitter#218)

`&` followed by space/newline was always lexed as bitwise-and; now it is a
block ampersand when no binary operator is valid (same check as unary minus).

Before: bar x, &
  (call (argument_list (identifier))) (ERROR)
After:
  (call (argument_list (identifier) (block_argument)))

Before: foo(a, &\n)
  (call (argument_list (identifier) (ERROR)))
After:
  (call (argument_list (identifier) (block_argument)))

Before: boo( & lambda {})
  (call (argument_list (ERROR) (call (identifier) (block))))
After:
  (call (argument_list (block_argument (call (identifier) (block)))))
…itter#203)

Body is now matched line by line; `=end` closes it only at column 0 and when
followed by whitespace or EOL, like Ruby.

Before: =begin\nhello\nworld =end\n=end
  (program (comment [0,0]-[2,10]) (ERROR [3,0]-[3,4]))
After:
  (program (comment [0,0]-[3,4]))
…#292)

`(char)lookahead` kept only the low byte, so U+8A00 became '\0', U+4F60 '`',
U+C77C '|'. Non-ASCII codepoints are now always identifier chars.

Before: :言
  (program (ERROR [0,0]-[0,1]) (identifier [0,1]-[0,4]))
After:
  (program (simple_symbol [0,0]-[0,4]))
…itter#256)

Like Ruby, a heredoc after an operand needs a command-argument position and a
space before `<<` (`foo <<EOS`); otherwise it is a left shift.

Before: "test"<<"ruby"
  (ERROR (string)) (heredoc_beginning) (heredoc_body ...)
After:
  (binary left: (string) right: (string))
…ee-sitter#281)

New `_unary_plus` external, emitted when `+` follows whitespace, precedes a
non-space and a binary operator would also fit; other `+` stay internal.

Before: something +x
  (binary left: (identifier) right: (identifier))
After:
  (call method: (identifier) arguments: (argument_list (unary operand: (identifier))))
…lause (tree-sitter#237)

A trailing hash shorthand suppresses the line break so the value can follow on
the next line; a `rescue` word there now gets a zero-width line break instead.

Before: begin\n  foo key:\nrescue => e\nend
  (begin (rescue_modifier body: (call ...) (ERROR) handler: (identifier)))
After:
  (begin (call ... (pair key: (hash_key_symbol))) (rescue variable: (exception_variable (identifier))))
…ree-sitter#275)

Like Ruby, `a..` at EOL is continued (`_no_line_break` hint, same trick as
hash shorthand); it stays endless before `)`, `]`, `end`, `rescue`...

Before: range = 0..\nrange = 1..2
  (assignment right: (range begin: (integer)))
  (assignment right: (range begin: (integer) end: (integer)))
After:
  (assignment right: (range begin: (integer)
    end: (assignment right: (range begin: (integer) end: (integer)))))
…ter#284)

`\d`, `\/`, `[[:alpha:]]\w`... are pattern syntax, not string escapes; regex
content is now split only around `#{}` interpolations.

Before: /[[[:alpha:]]\d]+/
  (regex (string_content [0,1]-[0,13]) (escape_sequence [0,13]-[0,15]) (string_content [0,15]-[0,17]))
After:
  (regex (string_content [0,1]-[0,17]))
… body (tree-sitter#148)

A heredoc opened inside another heredoc's body is queued in front of it and
starts at the end of that line, like Ruby; same-line heredocs stay FIFO.

Before: puts <<HERE\n  hello #{<<HERE}\n  world\nHERE\nHERE
  (heredoc_body [0,11]-[3,4] (heredoc_content) (interpolation (heredoc_beginning)) (heredoc_content) (heredoc_end))
  (heredoc_body [3,4]-[4,4] (heredoc_content) (heredoc_end))
After:
  (heredoc_body [0,11]-[4,4] (heredoc_content) (interpolation (heredoc_beginning))
    (heredoc_body [1,17]-[3,4] (heredoc_content) (heredoc_end))
    (heredoc_content) (heredoc_end))
…tter#241)

With combined injections (ERB), `#.*` ran across `%>` into every later tag;
the scanner now lexes same-line comments and ends them at the range end.

Before: <% end # if foo %>\n<% if baz %>...
  (comment [7,7]-[15,16])   # swallows every following tag
After:
  (comment [7,7]-[7,16]) ... (comment [15,7]-[15,16])
The `+@` exclusion was not needed (operator names never reach this path) and
made `puts +@count` a binary, unlike `puts -@count`.

Before: puts +@count
  (binary left: (identifier) right: (instance_variable))
After:
  (call method: (identifier) arguments: (argument_list (unary operand: (instance_variable))))
`##{x}` returned no token for the first `#`, so the internal lexer made the
rest of the line a comment (seen in prism/node_ext.rb); also keeps a trailing
`#` out of a nested heredoc started on that line.

Before: <<~MSG\n  #{a}##{b}\nMSG
  (heredoc_body (heredoc_content) (interpolation) (comment) (heredoc_content) (heredoc_end))
After:
  (heredoc_body (heredoc_content) (interpolation) (heredoc_content) (interpolation) (heredoc_content) (heredoc_end))
…r#241)

Corpus tests cannot set included ranges; this binding test covers the ERB case.

Before (8a77f59^): <% x = 1 # note %>\n<% y = 2 %> as two ranges
  (comment) runs into the second range and swallows `y = 2`
After:
  (program (assignment) (comment [0,9]-[0,16]) (assignment))
`(char)lookahead` in scan_short_interpolation aliased U+0140 to '@' and U+0124
to '$', and global_variable only allowed ASCII names, unlike `@ivar`.

Before: $é = 1 / "#ŀa" / "#$é"
  (ERROR (ERROR)) ... / (ERROR) (comment) / (string (string_content))
After:
  (assignment (global_variable) (integer)) / (string (string_content)) / (string (interpolation (global_variable)))
Content resuming after `#{}`/an escape started with look_for_heredoc_end set, so
`#{y}EOS` closed the heredoc; a serialized `mid_line` flag now tracks it. One
current_heredoc_index() helper replaces the two "first started heredoc" loops.

Before: x = <<EOS\n#{y}EOS\nEOS
  (heredoc_body (heredoc_content) (interpolation) (heredoc_end [1,4]-[1,7])) (constant)
After:
  (heredoc_body (heredoc_content) (interpolation) (heredoc_content) (heredoc_end [2,0]-[2,3]))
Compares tree-sitter ERROR/MISSING nodes and the start positions of constructs
with a 1:1 Prism mapping (defs, classes, blocks, heredocs, regexes, ranges...).

Before: no comparison with Ruby's own parser
After:  stdlib+gems 957/957 agree; ruby/spec 4434/4444 agree, 6 real bugs found (2 diffs are UTF-16 fixtures)
Matches the CLI 0.27.1 template; edition 2024 requires `unsafe extern` blocks.
Breaking for builders: Rust 1.85 or newer is now needed to compile the crate.

Before: edition = "2021"; extern "C" { fn tree_sitter_ruby() -> *const (); }
After:  edition = "2024"; unsafe extern "C" { fn tree_sitter_ruby() -> *const (); }
Same `__has_include` guard as the CLI 0.27.1 template; no behaviour change here
since src/scanner.c exists.

Before: // #include "../../src/scanner.c"
After:  // #if __has_include("../../src/scanner.c") ... // #endif
Matches the CLI 0.27.1 template; Cargo.lock resolves cc 1.2.66.

Before: cc = "1.1" (lock: 1.1.37)
After:  cc = "1.2" (lock: 1.2.66)
New `prism` job runs script/prism_diff.rb with a known-differences list; it fails
on any new difference, and when a listed file agrees again so the list only shrinks.

Before: no check against Ruby's own parser in CI
After:  ruby/spec @0816047: 4434 agree, 8 known (2 UTF-16 fixtures, 6 grammar bugs), exit 0
Block and hash shared the `{` token and LR merged both, so `a => b` resolved as
a hash pair; a dedicated `_block_open` decides at `{`, like Ruby. Parser shrinks.

Before: 1.times {\n  [0, 1] => [a, b]\n}  -> (call arguments: (argument_list (hash (pair ...))))
After:  (call block: (block (block_body (match_pattern ...)))); STATE_COUNT 6576 -> 6512
After `..`/`...` binary operators are only valid because the range may be
endless; Ruby reads an operand there, so `/` opens a regex, not a division.

Before: (//..//)  -> (range begin: (regex) (MISSING "/"))
After:            -> (range begin: (regex) end: (regex))
`=` was disabled as a percent delimiter because of the `%=` operator; `%=` now
stays the operator wherever a binary operator may follow, like Ruby.

Before: %=hey #{@ip}=.size  -> (ERROR); x %= 2 -> (operator_assignment)
After:  (call receiver: (string (string_content) (interpolation)) method: (identifier)); x %= 2 unchanged
Like Ruby's tLAMBEG, a new `_lambda_brace` token (scanner) marks the body brace,
so `x: { }` is not a hash default and `a=a() { a }` not a call's block. +10% states.

Before: -> x: { }  -> (ERROR (keyword_parameter value: (hash))); -> a=a() { a } -> ERROR
After:  (lambda parameters: (lambda_parameters ...) body: (block)); ruby/spec 4437 -> 4440 agree
They were excluded from parse-action because they produced errors; the
percent-literal and lambda-body fixes make them parse cleanly.

Before: 9 ruby/spec files excluded from parse-action
After:  6 excluded (invalid syntax, encodings, UTF-16 fixtures)
…ree-sitter#241)

After a crossed newline the token starts before it, so such comments went to the
`#.*` lexer; when the comment reaches an included range start, a zero-width line
break is emitted first and the comment is then lexed by the scanner.

Before: <% foo\n# note %>text<% bar %>  -> (comment) ends at byte 27, swallowing `bar`
After:  (comment [1,0]-[1,7]) "# note ", `bar` parsed; method chains after comments unchanged
`in`/`=>` had prec 100, so they shifted into endless def bodies and assignment
right-hand sides; like Ruby they now sit between `not` and `defined?` (PREC.MATCH).

Before: def ok? = x in Integer  -> (method body: (test_pattern ...)); a = b in c -> (assignment right: (test_pattern))
After:  (test_pattern value: (method ...) pattern: (constant)); (test_pattern value: (assignment) ...)
Bare keywords are `_primary`, so a binary operator looked valid after them and
`%=` lexed as an operator; a never-produced `_operand_next` marker tells the
scanner an operand comes next, like Ruby.

Before: return %=a=  -> (binary left: (return) (ERROR) right: (identifier)) (ERROR)
After:  (return (argument_list (string (string_content))))
Endless ranges made `<<` look like a valid shift after `..`; like Ruby (and the
existing `/` rule), an operand is read there, so `<<A` opens a heredoc.

Before: x = (1..<<A)\nbody\nA  -> (binary left: (range begin: (integer)) right: (constant)) (identifier) (constant)
After:  (range begin: (integer) end: (heredoc_beginning)) (heredoc_body (heredoc_content) (heredoc_end))
A space/tab/newline after `%` never set the delimiter, so the string ran to EOF;
it is now a delimiter wherever an operand is expected, and `p % a` stays binary.

Before: [% a ]\np % a  -> (array (string (string_content)))  (the string swallows `p % a`)
After:  (array (string (string_content))) (binary left: (identifier) right: (identifier))
`@a "x"`, `@a *b` and `self %(p)` parsed as method calls, so `@a <<"x"` opened a
heredoc; like Ruby, variables are operands, so these are binary operations.

Before: @A <<"x"  -> (call method: (instance_variable) arguments: (argument_list (heredoc_beginning))) (heredoc_body ...)
After:  (binary left: (instance_variable) right: (string (string_content))); prime.rb `self%(p)` -> (binary)
Nine near-identical `%x` cases, two `*`/`**` ladders and the `:"`/`:'` twins are
merged; destroy() calls reset(), a wrong comment is fixed, static/(void) added.

Before: src/scanner.c with 9 copied % cases and 2 copied star ladders (+151 lines)
After:  PERCENT_LITERALS table + scan_star_operator(); -80 net lines, byte-identical parses on 5401 files
CRLF heredoc, `%x(...)`, `"#$!"`, an unterminated string at EOF, and the
1024-byte state limit (171 nested literals); each fails if its branch breaks.

Before: 55 scanner lines never ran during `tree-sitter test`
After:  5 new corpus tests (333 parses); heredoc end word + trailing blanks left out (it differs from Ruby)

@clarifydoc clarifydoc left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this against ruby/spec, rails, discourse and chatwoot, comparing every tree with master. A lot of it is right:

  • ruby/spec files with errors drop from 11 to 4, and failing prism fixtures from 48 to 32;
  • def show = raise NotFound (#242) and ->(a; b) {} now parse;
  • @a -1.day is now subtraction (master called @a as a method), and return +"" if x is now a return;
  • return(1, 2) is rejected without breaking return(render json: x, status: :y).

One regression: a local variable followed by +1/-1 is now read as a method call with a unary argument:

[1].collect { |i| i +1 }   # master: binary (correct, i is a block parameter) → now: call i(+1)

ruby/spec's language/fixtures/precedence.rb tests exactly this (i +'1').

Two things for whoever reviews it:

  • Every return(x) now wraps its value in parenthesized_statements (114 files in that corpus change shape). That's correct, but consumers will see it.
  • The PR mixes the grammar work with CI, bindings, packaging, Swift/Python setup and new tags.scm/injections.scm queries, 49 files in all.

Splitting it, one topic per PR (endless command methods, lambda block-locals, return(…), the operand changes), would make the pieces that are clearly right much easier to merge.

Magikdidi24 and others added 10 commits October 25, 2026 21:11
go.sum resynced (go-tree-sitter 0.25 needs go 1.23) and package-lock regenerated;
eslint 10 no longer pulls in `globals`, which eslint-config-treesitter imports.

Before: go test -> missing go.sum entry; npm ci -> lock out of sync; eslint -> Cannot find package 'globals'
After:  cargo test, go test, tree-sitter test (333/333) and npm run lint pass
…previous one (tree-sitter#302)

Ruby 4.0 (Feature #20925). From upstream PR tree-sitter#303, with the char test made locale-independent
like the rest of this scanner (no <wctype.h>). Trees unchanged on 955 stdlib/gem files.

Before: c / || d  -> (identifier) (binary left: (identifier) (MISSING identifier) ...)
        e / and f -> (identifier) (call method: (identifier) arguments: (argument_list (identifier)))
After:  c / || d  -> (binary left: (identifier) right: (identifier))
        e / and f -> (binary left: (identifier) right: (identifier))
parser-test-action still defaults to Go 1.22, which go.mod no longer accepts.

Before: go get -t -> go.mod requires go >= 1.23 (running go 1.22.12; GOTOOLCHAIN=local)
After:  Go bindings build and test on Go 1.23
Gemfile.lock pins Prism, which otherwise came from whatever Ruby 4.0.x the runner had;
the CI prism job now runs RuboCop. Script output unchanged on ruby/spec and the stdlib.

Before: rubocop script/prism_diff.rb -> 69 offenses; Prism version = the runner's default gem
After:  bundle exec rubocop -> no offenses; prism 1.9.0, rubocop 1.92.1 from Gemfile.lock
Only comments that explain a non-obvious why are left. Generated parser unchanged.

Before: // with parameters, // Body can be a regular or "do" block, 3-line notes on one condition
After:  removed; the kept notes fit in one or two lines
A plain AllCops/Exclude replaced them, so CI linted the gems bundler-cache puts in vendor/bundle.

Before: bundle exec rubocop (CI) -> offenses in vendor/bundle/ruby/4.0.0/gems/...
After:  only Gemfile and script/ are inspected
Ruby reads an operand after these keywords, so the glued `<<` is not a shift (OPERAND_NEXT, like `..`).

Before: return<<A -> (binary left: (return) right: (constant)), body lines parsed as code
After:  return<<A -> (return (argument_list (heredoc_beginning))) (heredoc_body (heredoc_content) (heredoc_end))
Trailing blanks still closed the heredoc (`EOS \t`), Ruby keeps that line as content;
an end word right before EOF was read as content.

Before: <<A, a, "A ", A -> closes on "A ", the last A is a (constant); "EOS" at EOF -> (heredoc_end [2, 3] - [2, 3])
After:  closes on the bare A; "EOS" at EOF -> (heredoc_end [2, 0] - [2, 3])
npm, crates.io and PyPI names belong to upstream and the fork has no tokens for them.

Before: pushing a tag -> github + npm + crates + pypi jobs, the last three fail
After:  pushing a tag -> GitHub release with the wasm and the full source tarball
Minor bump: Rust >= 1.85 (edition 2024) and `call.operator` in node-types.json changed since 0.23.1.

Before: version 0.23.1 in tree-sitter.json, package.json, Cargo.toml, pyproject.toml, CMakeLists.txt, Makefile
After:  0.24.0 everywhere (bumped with `tree-sitter version 0.24.0`, lockfiles synced)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants