refactor: migrate syntax highlighting to Tree-sitter

This commit is contained in:
Valdemar Lorenzen
2026-10-09 11:15:17 +02:00
parent 7c1e4c3ca0
commit e5d545a4ca
5 changed files with 2079 additions and 672 deletions
+4 -4
View File
@@ -205,7 +205,7 @@ One crate, seven modules, no workspace or terminal-coordinate types in the editi
- `editor`: owns buffer lifecycle and per-buffer text, saved state, primary/selection set, modes, search, and undo/redo; session-wide clipboard policy, independent registers and owned-system fragment metadata, semantic native repeat, action metadata, command-palette and file-picker focus/dispatch, key notation, leader/count prefixes and binding resolution, modal paste policy, saving, command ownership, capabilities, and validated edit effects. Its low-level interface returns an `Invocation` when an external command is selected.
- `files`: owns the gitignore-aware directory/file catalog, authoritative OS paths, compact tree projection, query filtering, browse/search focus, expansion/selection restoration, navigation, and scan/error state shared by floating and docked presentations.
- `clipboard`: owns native plain-text provider selection and bounded subprocess I/O; it never computes selections or editing offsets.
- `syntax`: owns bundled grammar detection, semantic byte spans, multiline parser checkpoints, and changed-suffix highlighting; it has no terminal color/geometry or editing dependency.
- `syntax`: owns Tree-sitter grammar detection, per-buffer incremental trees, bundled highlight/local/injection queries, and semantic byte spans; it has no terminal color/geometry or editing dependency.
- `session`: `EditorSession` owns the editor and its Ruby hosts together. It loads configuration once, registers eligible plugins before installing shared bindings, executes selected commands against the active buffer, and retires both host and command ownership on invocation or effect-validation failure.
- `tui`: translates terminal events to session input, renders read-only active-buffer state, keeps frontend-owned viewports and syntax caches by stable buffer ID, and advertises `text` and `styled_text` capabilities. It maps semantic tokens to terminal colors and has no plugin lifecycle callback, host storage, or paste acceptance policy.
- `ruby`: evaluates configuration and runs isolated plugin hosts. Protocol, deadlines, and process cleanup stay inside this module.
@@ -228,11 +228,11 @@ The buffer viewport includes a fixed left gutter with right-aligned hybrid line
### Syntax highlighting
Buffers highlight automatically using [Syntect's bundled Sublime grammars](https://docs.rs/syntect/5.3.0/syntect/). Detection uses the supplied filename/extension and the buffer's own first line/shebang, without rereading the file from disk. Tested languages include Rust, Ruby, Python, JSON, shell, Makefile, and Markdown; other bundled grammars are available. Unknown and Plain Text buffers keep the terminal's normal foreground. The detected language appears in status, and saving under a different extension redetects it.
Buffers highlight automatically using [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) and compiled-in grammar/query packages. The selected language set is Rust, Ruby, Python, JSON, Bash, Makefile, Markdown, JavaScript/JSX, TypeScript/TSX, HTML, CSS, YAML, TOML, C, C++, and Go. Detection uses filename/extension or the in-memory first-line shebang, including common special filenames such as Makefile, Gemfile, and Cargo.lock; no file reread occurs. Unsupported formats use the normal foreground. This migration intentionally replaces the broader former grammar bundle: languages outside this explicit set, such as Lua/Haskell/Scala/Perl, are plain text until added. The detected language appears in status, and save-as redetects it.
Colors use the terminal palette rather than a fixed background/theme: comments are dark gray, strings green, keywords magenta, numbers yellow, types/markup cyan, and functions light blue. Multiline/nested comments and strings retain parser scope across lines, including when their opening is above the viewport. Highlighting styles whole graphemes and expanded tab cells; partially clipped wide characters keep the existing blank-placeholder behavior. Search, Visual selections, and secondary carets paint afterward and retain their foreground/background precedence.
Colors retain the terminal palette: comments dark gray, strings green, keywords magenta, numbers yellow, types/markup cyan, and functions light blue. Grammar queries handle multiline/nested constructs, even when their opening is above the viewport. Markdown uses its inline grammar and named fenced-code injections; HTML script/style content uses JavaScript/CSS. Known tagged-template and raw-string injections use the same selected grammar registry; unknown injected languages retain host highlighting or plain text. Highlighting styles whole graphemes and expanded tab cells; partially clipped wide placeholders remain unchanged. Search, Visual selections, and secondary carets retain foreground/background precedence.
Each buffer has a frontend-owned cache. Text revisions distinguish edits/plugin effects/undo/redo from cursor and explorer changes; unchanged revision/path skips comparison and parsing. Changed text resumes at the first changed line's cached parser/scope state and reparses the suffix, preserving prefix work. Buffer switching restores its cache, and closing retires it. Real parser/scope errors clear syntax spans, retain readable/editable plain text, and report a status error. No custom lexer, runtime grammar downloads, or editor-offset changes are involved.
Each buffer has a frontend-owned parser/tree cache. Unchanged revision/path skips comparison and parsing; unchanged text at a later revision also avoids parsing. Actual text changes produce a minimal UTF-8-safe byte edit with newline-aware Tree-sitter Points, edit the prior tree, then parse using that tree for incremental reuse. Grammar changes reset the tree, including TypeScript versus TSX despite their shared display name. Native highlight/local/injection queries resolve capture precedence and scoped definitions into nonoverlapping original-line spans; unchanged injected trees are reused. Buffer switching restores its cache, and closing retires it. Incomplete text and grammar ERROR nodes remain highlightable; genuine language/query/parser failures expose a status error and readable plain text. Full-tree highlight queries still run after incremental parsing—a deliberate ceiling to revisit if measured costly. Syntect, its scope checkpoints, and its dependencies are removed; there is no second baseline engine, runtime grammar download, or LSP scaffold.
### Core feature registry