Repository navigation
add some important features from ruby > 3.0 and solve some issue - #282
Magikdidi24 wants to merge 73 commits into
Conversation
Magikdidi24
commented
Oct 20, 2025
- shadows variables
- for range with begin only
- functions ruby >3.0
- bad return
| a = 42 | ||
| -> { | ||
| # This is also another a. | ||
| use a |
There was a problem hiding this comment.
Could you elaborate why this not the same as a on line 2771?
There was a problem hiding this comment.
This is a mistake — it’s the same. I just used my own test case in my project and forgot to remove the comment.
0738b52 to
65bb643
Compare
Committed src/ was out of sync with grammar.js (stale conflict entry). Before (src/grammar.json conflicts): [["_expression","_chained_command_call"], ..., ["_statement","call"]] After: [["_expression","_chained_command_call"], ...]
Expected tree listed a (comment) node that is not in the source.
Before:
body: (block
(comment)
body: (block_body
After:
body: (block
body: (block_body
…e-sitter#290, tree-sitter#244, tree-sitter#218) `&` followed by space/newline was always lexed as bitwise-and; now it is a block ampersand when no binary operator is valid (same check as unary minus). Before: bar x, & (call (argument_list (identifier))) (ERROR) After: (call (argument_list (identifier) (block_argument))) Before: foo(a, &\n) (call (argument_list (identifier) (ERROR))) After: (call (argument_list (identifier) (block_argument))) Before: boo( & lambda {}) (call (argument_list (ERROR) (call (identifier) (block)))) After: (call (argument_list (block_argument (call (identifier) (block)))))
…itter#203) Body is now matched line by line; `=end` closes it only at column 0 and when followed by whitespace or EOL, like Ruby. Before: =begin\nhello\nworld =end\n=end (program (comment [0,0]-[2,10]) (ERROR [3,0]-[3,4])) After: (program (comment [0,0]-[3,4]))
…#292) `(char)lookahead` kept only the low byte, so U+8A00 became '\0', U+4F60 '`', U+C77C '|'. Non-ASCII codepoints are now always identifier chars. Before: :言 (program (ERROR [0,0]-[0,1]) (identifier [0,1]-[0,4])) After: (program (simple_symbol [0,0]-[0,4]))
…itter#256) Like Ruby, a heredoc after an operand needs a command-argument position and a space before `<<` (`foo <<EOS`); otherwise it is a left shift. Before: "test"<<"ruby" (ERROR (string)) (heredoc_beginning) (heredoc_body ...) After: (binary left: (string) right: (string))
…ee-sitter#281) New `_unary_plus` external, emitted when `+` follows whitespace, precedes a non-space and a binary operator would also fit; other `+` stay internal. Before: something +x (binary left: (identifier) right: (identifier)) After: (call method: (identifier) arguments: (argument_list (unary operand: (identifier))))
…lause (tree-sitter#237) A trailing hash shorthand suppresses the line break so the value can follow on the next line; a `rescue` word there now gets a zero-width line break instead. Before: begin\n foo key:\nrescue => e\nend (begin (rescue_modifier body: (call ...) (ERROR) handler: (identifier))) After: (begin (call ... (pair key: (hash_key_symbol))) (rescue variable: (exception_variable (identifier))))
…ree-sitter#275) Like Ruby, `a..` at EOL is continued (`_no_line_break` hint, same trick as hash shorthand); it stays endless before `)`, `]`, `end`, `rescue`... Before: range = 0..\nrange = 1..2 (assignment right: (range begin: (integer))) (assignment right: (range begin: (integer) end: (integer))) After: (assignment right: (range begin: (integer) end: (assignment right: (range begin: (integer) end: (integer)))))
…ter#284) `\d`, `\/`, `[[:alpha:]]\w`... are pattern syntax, not string escapes; regex content is now split only around `#{}` interpolations. Before: /[[[:alpha:]]\d]+/ (regex (string_content [0,1]-[0,13]) (escape_sequence [0,13]-[0,15]) (string_content [0,15]-[0,17])) After: (regex (string_content [0,1]-[0,17]))
… body (tree-sitter#148) A heredoc opened inside another heredoc's body is queued in front of it and starts at the end of that line, like Ruby; same-line heredocs stay FIFO. Before: puts <<HERE\n hello #{<<HERE}\n world\nHERE\nHERE (heredoc_body [0,11]-[3,4] (heredoc_content) (interpolation (heredoc_beginning)) (heredoc_content) (heredoc_end)) (heredoc_body [3,4]-[4,4] (heredoc_content) (heredoc_end)) After: (heredoc_body [0,11]-[4,4] (heredoc_content) (interpolation (heredoc_beginning)) (heredoc_body [1,17]-[3,4] (heredoc_content) (heredoc_end)) (heredoc_content) (heredoc_end))
…tter#241) With combined injections (ERB), `#.*` ran across `%>` into every later tag; the scanner now lexes same-line comments and ends them at the range end. Before: <% end # if foo %>\n<% if baz %>... (comment [7,7]-[15,16]) # swallows every following tag After: (comment [7,7]-[7,16]) ... (comment [15,7]-[15,16])
The `+@` exclusion was not needed (operator names never reach this path) and made `puts +@count` a binary, unlike `puts -@count`. Before: puts +@count (binary left: (identifier) right: (instance_variable)) After: (call method: (identifier) arguments: (argument_list (unary operand: (instance_variable))))
`##{x}` returned no token for the first `#`, so the internal lexer made the
rest of the line a comment (seen in prism/node_ext.rb); also keeps a trailing
`#` out of a nested heredoc started on that line.
Before: <<~MSG\n #{a}##{b}\nMSG
(heredoc_body (heredoc_content) (interpolation) (comment) (heredoc_content) (heredoc_end))
After:
(heredoc_body (heredoc_content) (interpolation) (heredoc_content) (interpolation) (heredoc_content) (heredoc_end))
…r#241) Corpus tests cannot set included ranges; this binding test covers the ERB case. Before (8a77f59^): <% x = 1 # note %>\n<% y = 2 %> as two ranges (comment) runs into the second range and swallows `y = 2` After: (program (assignment) (comment [0,9]-[0,16]) (assignment))
`(char)lookahead` in scan_short_interpolation aliased U+0140 to '@' and U+0124 to '$', and global_variable only allowed ASCII names, unlike `@ivar`. Before: $é = 1 / "#ŀa" / "#$é" (ERROR (ERROR)) ... / (ERROR) (comment) / (string (string_content)) After: (assignment (global_variable) (integer)) / (string (string_content)) / (string (interpolation (global_variable)))
Content resuming after `#{}`/an escape started with look_for_heredoc_end set, so
`#{y}EOS` closed the heredoc; a serialized `mid_line` flag now tracks it. One
current_heredoc_index() helper replaces the two "first started heredoc" loops.
Before: x = <<EOS\n#{y}EOS\nEOS
(heredoc_body (heredoc_content) (interpolation) (heredoc_end [1,4]-[1,7])) (constant)
After:
(heredoc_body (heredoc_content) (interpolation) (heredoc_content) (heredoc_end [2,0]-[2,3]))
Compares tree-sitter ERROR/MISSING nodes and the start positions of constructs with a 1:1 Prism mapping (defs, classes, blocks, heredocs, regexes, ranges...). Before: no comparison with Ruby's own parser After: stdlib+gems 957/957 agree; ruby/spec 4434/4444 agree, 6 real bugs found (2 diffs are UTF-16 fixtures)
Matches the CLI 0.27.1 template; edition 2024 requires `unsafe extern` blocks.
Breaking for builders: Rust 1.85 or newer is now needed to compile the crate.
Before: edition = "2021"; extern "C" { fn tree_sitter_ruby() -> *const (); }
After: edition = "2024"; unsafe extern "C" { fn tree_sitter_ruby() -> *const (); }
Same `__has_include` guard as the CLI 0.27.1 template; no behaviour change here
since src/scanner.c exists.
Before: // #include "../../src/scanner.c"
After: // #if __has_include("../../src/scanner.c") ... // #endif
Matches the CLI 0.27.1 template; Cargo.lock resolves cc 1.2.66. Before: cc = "1.1" (lock: 1.1.37) After: cc = "1.2" (lock: 1.2.66)
New `prism` job runs script/prism_diff.rb with a known-differences list; it fails on any new difference, and when a listed file agrees again so the list only shrinks. Before: no check against Ruby's own parser in CI After: ruby/spec @0816047: 4434 agree, 8 known (2 UTF-16 fixtures, 6 grammar bugs), exit 0
Block and hash shared the `{` token and LR merged both, so `a => b` resolved as
a hash pair; a dedicated `_block_open` decides at `{`, like Ruby. Parser shrinks.
Before: 1.times {\n [0, 1] => [a, b]\n} -> (call arguments: (argument_list (hash (pair ...))))
After: (call block: (block (block_body (match_pattern ...)))); STATE_COUNT 6576 -> 6512
After `..`/`...` binary operators are only valid because the range may be endless; Ruby reads an operand there, so `/` opens a regex, not a division. Before: (//..//) -> (range begin: (regex) (MISSING "/")) After: -> (range begin: (regex) end: (regex))
`=` was disabled as a percent delimiter because of the `%=` operator; `%=` now
stays the operator wherever a binary operator may follow, like Ruby.
Before: %=hey #{@ip}=.size -> (ERROR); x %= 2 -> (operator_assignment)
After: (call receiver: (string (string_content) (interpolation)) method: (identifier)); x %= 2 unchanged
Like Ruby's tLAMBEG, a new `_lambda_brace` token (scanner) marks the body brace,
so `x: { }` is not a hash default and `a=a() { a }` not a call's block. +10% states.
Before: -> x: { } -> (ERROR (keyword_parameter value: (hash))); -> a=a() { a } -> ERROR
After: (lambda parameters: (lambda_parameters ...) body: (block)); ruby/spec 4437 -> 4440 agree
They were excluded from parse-action because they produced errors; the percent-literal and lambda-body fixes make them parse cleanly. Before: 9 ruby/spec files excluded from parse-action After: 6 excluded (invalid syntax, encodings, UTF-16 fixtures)
…ree-sitter#241) After a crossed newline the token starts before it, so such comments went to the `#.*` lexer; when the comment reaches an included range start, a zero-width line break is emitted first and the comment is then lexed by the scanner. Before: <% foo\n# note %>text<% bar %> -> (comment) ends at byte 27, swallowing `bar` After: (comment [1,0]-[1,7]) "# note ", `bar` parsed; method chains after comments unchanged
`in`/`=>` had prec 100, so they shifted into endless def bodies and assignment right-hand sides; like Ruby they now sit between `not` and `defined?` (PREC.MATCH). Before: def ok? = x in Integer -> (method body: (test_pattern ...)); a = b in c -> (assignment right: (test_pattern)) After: (test_pattern value: (method ...) pattern: (constant)); (test_pattern value: (assignment) ...)
Bare keywords are `_primary`, so a binary operator looked valid after them and `%=` lexed as an operator; a never-produced `_operand_next` marker tells the scanner an operand comes next, like Ruby. Before: return %=a= -> (binary left: (return) (ERROR) right: (identifier)) (ERROR) After: (return (argument_list (string (string_content))))
Endless ranges made `<<` look like a valid shift after `..`; like Ruby (and the existing `/` rule), an operand is read there, so `<<A` opens a heredoc. Before: x = (1..<<A)\nbody\nA -> (binary left: (range begin: (integer)) right: (constant)) (identifier) (constant) After: (range begin: (integer) end: (heredoc_beginning)) (heredoc_body (heredoc_content) (heredoc_end))
A space/tab/newline after `%` never set the delimiter, so the string ran to EOF; it is now a delimiter wherever an operand is expected, and `p % a` stays binary. Before: [% a ]\np % a -> (array (string (string_content))) (the string swallows `p % a`) After: (array (string (string_content))) (binary left: (identifier) right: (identifier))
`@a "x"`, `@a *b` and `self %(p)` parsed as method calls, so `@a <<"x"` opened a heredoc; like Ruby, variables are operands, so these are binary operations. Before: @A <<"x" -> (call method: (instance_variable) arguments: (argument_list (heredoc_beginning))) (heredoc_body ...) After: (binary left: (instance_variable) right: (string (string_content))); prime.rb `self%(p)` -> (binary)
Nine near-identical `%x` cases, two `*`/`**` ladders and the `:"`/`:'` twins are merged; destroy() calls reset(), a wrong comment is fixed, static/(void) added. Before: src/scanner.c with 9 copied % cases and 2 copied star ladders (+151 lines) After: PERCENT_LITERALS table + scan_star_operator(); -80 net lines, byte-identical parses on 5401 files
CRLF heredoc, `%x(...)`, `"#$!"`, an unterminated string at EOF, and the 1024-byte state limit (171 nested literals); each fails if its branch breaks. Before: 55 scanner lines never ran during `tree-sitter test` After: 5 new corpus tests (333 parses); heredoc end word + trailing blanks left out (it differs from Ruby)
clarifydoc
left a comment
There was a problem hiding this comment.
I ran this against ruby/spec, rails, discourse and chatwoot, comparing every tree with master. A lot of it is right:
- ruby/spec files with errors drop from 11 to 4, and failing prism fixtures from 48 to 32;
def show = raise NotFound(#242) and->(a; b) {}now parse;@a -1.dayis now subtraction (master called@aas a method), andreturn +"" if xis now a return;return(1, 2)is rejected without breakingreturn(render json: x, status: :y).
One regression: a local variable followed by +1/-1 is now read as a method call with a unary argument:
[1].collect { |i| i +1 } # master: binary (correct, i is a block parameter) → now: call i(+1)ruby/spec's language/fixtures/precedence.rb tests exactly this (i +'1').
Two things for whoever reviews it:
- Every
return(x)now wraps its value inparenthesized_statements(114 files in that corpus change shape). That's correct, but consumers will see it. - The PR mixes the grammar work with CI, bindings, packaging, Swift/Python setup and new
tags.scm/injections.scmqueries, 49 files in all.
Splitting it, one topic per PR (endless command methods, lambda block-locals, return(…), the operand changes), would make the pieces that are clearly right much easier to merge.
go.sum resynced (go-tree-sitter 0.25 needs go 1.23) and package-lock regenerated; eslint 10 no longer pulls in `globals`, which eslint-config-treesitter imports. Before: go test -> missing go.sum entry; npm ci -> lock out of sync; eslint -> Cannot find package 'globals' After: cargo test, go test, tree-sitter test (333/333) and npm run lint pass
…previous one (tree-sitter#302) Ruby 4.0 (Feature #20925). From upstream PR tree-sitter#303, with the char test made locale-independent like the rest of this scanner (no <wctype.h>). Trees unchanged on 955 stdlib/gem files. Before: c / || d -> (identifier) (binary left: (identifier) (MISSING identifier) ...) e / and f -> (identifier) (call method: (identifier) arguments: (argument_list (identifier))) After: c / || d -> (binary left: (identifier) right: (identifier)) e / and f -> (binary left: (identifier) right: (identifier))
parser-test-action still defaults to Go 1.22, which go.mod no longer accepts. Before: go get -t -> go.mod requires go >= 1.23 (running go 1.22.12; GOTOOLCHAIN=local) After: Go bindings build and test on Go 1.23
Gemfile.lock pins Prism, which otherwise came from whatever Ruby 4.0.x the runner had; the CI prism job now runs RuboCop. Script output unchanged on ruby/spec and the stdlib. Before: rubocop script/prism_diff.rb -> 69 offenses; Prism version = the runner's default gem After: bundle exec rubocop -> no offenses; prism 1.9.0, rubocop 1.92.1 from Gemfile.lock
Only comments that explain a non-obvious why are left. Generated parser unchanged. Before: // with parameters, // Body can be a regular or "do" block, 3-line notes on one condition After: removed; the kept notes fit in one or two lines
A plain AllCops/Exclude replaced them, so CI linted the gems bundler-cache puts in vendor/bundle. Before: bundle exec rubocop (CI) -> offenses in vendor/bundle/ruby/4.0.0/gems/... After: only Gemfile and script/ are inspected
Ruby reads an operand after these keywords, so the glued `<<` is not a shift (OPERAND_NEXT, like `..`). Before: return<<A -> (binary left: (return) right: (constant)), body lines parsed as code After: return<<A -> (return (argument_list (heredoc_beginning))) (heredoc_body (heredoc_content) (heredoc_end))
Trailing blanks still closed the heredoc (`EOS \t`), Ruby keeps that line as content; an end word right before EOF was read as content. Before: <<A, a, "A ", A -> closes on "A ", the last A is a (constant); "EOS" at EOF -> (heredoc_end [2, 3] - [2, 3]) After: closes on the bare A; "EOS" at EOF -> (heredoc_end [2, 0] - [2, 3])
npm, crates.io and PyPI names belong to upstream and the fork has no tokens for them. Before: pushing a tag -> github + npm + crates + pypi jobs, the last three fail After: pushing a tag -> GitHub release with the wasm and the full source tarball
Minor bump: Rust >= 1.85 (edition 2024) and `call.operator` in node-types.json changed since 0.23.1. Before: version 0.23.1 in tree-sitter.json, package.json, Cargo.toml, pyproject.toml, CMakeLists.txt, Makefile After: 0.24.0 everywhere (bumped with `tree-sitter version 0.24.0`, lockfiles synced)