feat: enable LSP semantic token highlighting by default

This commit is contained in:
2026-10-09 22:17:30 +02:00
parent 548711980c
commit d418d670ee
13 changed files with 1714 additions and 84 deletions
+7 -4
View File
@@ -11,9 +11,9 @@ One crate; no workspace or terminal-coordinate types in the editing backend:
- `editor`: owns buffer lifecycle and per-buffer text, saved state, primary/selection set, modes, search, and undo/redo; session-wide clipboard policy, independent registers and owned-system fragment metadata, semantic native repeat, action metadata, command-palette and file-picker focus/dispatch, key notation, leader/count prefixes and binding resolution, modal paste policy, saving, command ownership, capabilities, and validated edit effects. Its low-level interface returns an `Invocation` when an external command is selected.
- `files`: owns the gitignore-aware directory/file catalog, authoritative OS paths, compact tree projection, query filtering, browse/search focus, expansion/selection restoration, navigation, and scan/error state shared by floating and docked presentations.
- `clipboard`: owns native plain-text provider selection and bounded subprocess I/O; it never computes selections or editing offsets.
- `syntax`: owns Tree-sitter grammar detection, per-buffer incremental trees, bundled highlight/local/injection queries, and semantic byte spans; it has no terminal color/geometry or editing dependency.
- `plugin`: owns generic document snapshots/events, zero-based grapheme positions, diagnostics, and versioned asynchronous update contracts. It has no LSP protocol types.
- `lsp`: owns opt-in native server workers, bounded framed JSON-RPC, synchronization/capability negotiation, settings, UTF-16/file-URI conversion, and definition/hover results. Its private configuration resolver supplies presets without restricting language IDs.
- `syntax`: owns Tree-sitter grammar detection, per-buffer incremental trees, bundled highlight/local/injection queries, and syntax byte spans; it has no terminal color/geometry or editing dependency.
- `plugin`: owns generic document snapshots/events, zero-based grapheme positions, diagnostics, frontend-independent semantic token kinds/modifiers, and versioned asynchronous update contracts. Semantic token updates are native-only, not part of the Ruby wire protocol; it has no LSP protocol types.
- `lsp`: owns opt-in native server workers, bounded framed JSON-RPC, synchronization/capability negotiation, settings, UTF-16/file-URI conversion, definition/hover results, and semantic token requests/legend decoding. Its private configuration resolver supplies presets without restricting language IDs.
- `session`: `EditorSession` owns the editor, native LSP, and Ruby hosts together. It loads configuration once, registers eligible plugins before installing shared bindings, tracks document open/change/save/close events, polls native and plugin results, validates versioned diagnostics/navigation, and retires failed plugin ownership.
- `tui`: translates terminal events to session input, renders read-only active-buffer state, keeps frontend-owned viewports and syntax caches by stable buffer ID, and advertises `text`, `styled_text`, and `diagnostics` capabilities. It maps semantic tokens and diagnostics to terminal styling and has no server/protocol logic or paste acceptance policy.
- `ruby`: evaluates configuration and runs isolated plugin hosts. Protocol, deadlines, and process cleanup stay inside this module.
@@ -30,6 +30,7 @@ Frontends inspect `leader_pending()` for the cached display of a pending leader
Frontends inspect `buffer_id()`, `buffer_count()`, `has_buffer(id)`, and the ascending-ID `buffers()` iterator of borrowed path/dirty/active metadata. Existing text, cursor, selection, search, and mode methods describe only the active buffer.
Frontends inspect `file_picker()` for query, presentation, view, explorer-local INSERT/filter subfocus, selection, and errors, and iterate borrowed `FileRow` metadata (actual path, display label, directory flag, visible depth, expansion). Dock side is a logical preference. They allocate rectangles, add folder suffixes/colors, and render; hierarchy, filtering, parent/child navigation, focus transitions, and selected-file opening remain backend-owned.
Frontends inspect `diagnostics()` and `diagnostic_at_cursor()` for validated active-buffer annotations. Diagnostics are owned by core LSP or a plugin and buffer revision; text changes clear them. Native server retirement removes only that server's contribution, preserving sibling diagnostics. The TUI underlines annotated text, colors affected gutter numbers by severity, and displays the primary caret's diagnostic message without replacing search/selection backgrounds.
Frontends borrow `semantic_tokens()` for sorted, single-line grapheme spans from the active buffer. Immutable `Arc` slices are shared by native LSP and buffer storage; painting does not copy tokens. Text/path changes discard overlays, and native updates validate the whole applicable annotation batch before mutation. The TUI merges semantic styling into its existing glyph pass above syntax foregrounds but below diagnostic/search/selection/caret styling.
Frontends inspect `hover()` for optional read-only content and logical-line/grapheme scroll offsets. Popup request eligibility, dismissal, and scrolling stay editor/session-owned; the TUI clips the panel inside the document workspace without changing the primary cursor or viewport.
## Terminal lifecycle and layout
@@ -42,7 +43,7 @@ The buffer viewport includes a fixed left gutter with right-aligned hybrid line
## Syntax caches
Each buffer has a frontend-owned parser/tree cache. Unchanged revision/path skips comparison and parsing; unchanged text at a later revision also avoids parsing. Actual text changes produce a minimal UTF-8-safe byte edit with newline-aware Tree-sitter Points, edit the prior tree, then parse using that tree for incremental reuse. Grammar changes reset the tree, including TypeScript versus TSX despite their shared display name. Native highlight/local/injection queries resolve capture precedence and scoped definitions into nonoverlapping original-line spans; unchanged injected trees are reused. Buffer switching restores its cache, and closing retires it. Incomplete text and grammar ERROR nodes remain highlightable; genuine language/query/parser failures expose a status error and readable plain text. Full-tree highlight queries still run after incremental parsing—a deliberate ceiling to revisit if measured costly. There is no second highlighting engine or runtime grammar download.
Each buffer has a frontend-owned parser/tree cache. Unchanged revision/path skips comparison and parsing; unchanged text at a later revision also avoids parsing. Actual text changes produce a minimal UTF-8-safe byte edit with newline-aware Tree-sitter Points, edit the prior tree, then parse using that tree for incremental reuse. Grammar changes reset the tree, including TypeScript versus TSX despite their shared display name. Native highlight/local/injection queries resolve capture precedence and scoped definitions into nonoverlapping original-line spans; unchanged injected trees are reused. Buffer switching restores its cache, and closing retires it. Incomplete text and grammar ERROR nodes remain highlightable; genuine language/query/parser failures expose a status error and readable plain text. Full-tree highlight queries still run after incremental parsing—a deliberate ceiling to revisit if measured costly. There is no local semantic inference engine or runtime grammar download.
## Clipboard providers
@@ -54,6 +55,8 @@ The native module initializes each server, sends `initialization_options` as `in
Language configurations can own an ordered server stack. The first matching language wins extension overlap; all its matching servers receive document events, while the first handles hover/definition. The Python preset uses ty then Ruff. LSP merges each worker's latest diagnostic snapshot in assignment order before emitting the unchanged core-owned update. Empty publications/retirement remove only their source; revision/path/root changes and close discard all contributions. Worker snapshots retain identity so queued pre-rename diagnostics cannot attach to a new path with the same text revision. Combined diagnostic message bytes remain bounded to 4 MiB; an overflowing publication retires its worker instead of discarding healthy sibling annotations. Workers are stored in config/root order, avoiding per-poll key cloning/sorting. Shutdown flushes the exit notification and waits for peer termination within the existing bounded grace period before process cleanup.
Semantic tokens belong only to the first assigned server. Initialization advertises standard token types/modifiers, relative encoding, full/range requests, and server refresh support; deltas, overlapping/multiline tokens, and dynamic registration are not advertised. Each buffer has at most one request in flight; changes/refreshes coalesce into the latest snapshot. Full requests are preferred, with whole-document range fallback. A linear UTF-16 decoder validates raw coordinates and legend indices before outward grapheme rounding; overlong ends truncate at EOL, unknown types and invisible spans fall back to syntax. Both wire frames and decoded spans are bounded to 4 MiB. Snapshot identity rejects stale/renamed/closed replies before decoding. Invalid semantic replies clear the overlay and report an error without retiring diagnostics/navigation; primary retirement clears its overlays, and sibling retirement leaves them intact.
## Ruby action catalog
Ruby builds `config.core` from the Rust action catalog before loading configuration or plugin scripts. Dotted identifiers create nested namespaces automatically; each namespace and Action value is frozen. Action names are not duplicated in Ruby, so Rust catalog additions/renames immediately change the Ruby interface without code generation or handwritten exports.
+6 -2
View File
@@ -179,7 +179,7 @@ config.lsp["python"] = {
**uv remains optional project tooling.** Run project setup/synchronization yourself; [ty discovers an existing `.venv`](https://docs.astral.sh/ty/reference/configuration/#python) without a uv launcher. Vamp neither runs `uv sync` nor changes dependencies/lockfiles. The default ty preset sets `experimental.useUv: "off"` so inherited `TY_UV` cannot enable ty's [experimental environment-management integration](https://docs.astral.sh/ty/reference/editor-settings/). Ruff's initialization options default to `{}`. When replacing ty's initialization options, retain that flag if you want the same policy.
Servers start lazily only when a matching buffer opens. Extension matching is case-sensitive; if enabled languages overlap, the first language in sorted key order wins. All matching servers in that language run, and the first handles hover/definition and the diagnostic-status command without fallback. Each server/project root gets one worker, with at most 16 configured servers and 16 server/project keys; Python uses two slots. Start vamp from your project root: its startup working directory is the workspace; automatic root discovery is not implemented.
Servers start lazily only when a matching buffer opens. Extension matching is case-sensitive; if enabled languages overlap, the first language in sorted key order wins. All matching servers in that language run, and the first handles hover/definition, semantic highlighting, and the diagnostic-status command without fallback to sibling servers. Each server/project root gets one worker, with at most 16 configured servers and 16 server/project keys; Python uses two slots. Start vamp from your project root: its startup working directory is the workspace; automatic root discovery is not implemented.
See [native LSP architecture](architecture.md#native-lsp) for synchronization and worker behavior.
@@ -195,4 +195,8 @@ Diagnostics appear as severity-colored gutter numbers and text underlines; the p
Hover accepts plaintext/Markdown MarkupContent and string/code MarkedString forms, including arrays. Multiline content is preserved; Markdown is shown as plain text, not rendered markup. Horizontal scrolling exposes long lines without splitting graphemes. Missing/unsupported/invalid hover results report a message without changing text, selections, or undo history. Content is bounded to 4 MiB and terminal controls are sanitized.
Native LSP currently provides synchronization, diagnostics, go-to-definition, and hover—not completion, rename, formatting, code actions, workspace edits, or semantic highlighting. Tree-sitter remains the syntax highlighter.
Semantic highlighting is **enabled automatically** when the first server advertises [LSP semantic tokens](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens). No additional configuration or binding is needed. Full-document requests are preferred; range-only servers receive a whole-document range request. Token colors override Tree-sitter where recognized; unsupported servers, unknown custom token types, missing tokens, and failed replies retain syntax highlighting. Text/path changes clear old semantic colors immediately, and stale replies cannot restore them. Same-text refreshes keep the current overlay until the replacement arrives.
rust-analyzer and ty support semantic tokens. Solargraph 0.61.0 and Ruff do not; Ruby therefore retains Tree-sitter colors with the Solargraph preset, while Python uses ty's tokens and Ruff's diagnostics. Declaration/definition modifiers use bold, deprecated symbols use strikethrough, and readonly variables/properties use yellow. Diagnostics, search, selections, and carets retain their existing styling precedence.
Native LSP currently provides synchronization, diagnostics, go-to-definition, hover, and semantic highlighting—not completion, rename, formatting, code actions, or workspace edits. Semantic token deltas, dynamic registration, and partial-result streaming are not implemented.
+1
View File
@@ -37,6 +37,7 @@ Rust tests live alongside their modules; native LSP integration checks are in [`
- Unix terminal-hook checks use isolated child processes and real PTYs to keep process-global hooks out of the parallel runner. They cover cleanup on entry failure, normal return, I/O errors, and panics, including original panic-payload propagation.
- Native LSP integration checks launch framed-protocol fixture subprocesses, covering initialization/settings, opt-in startup, synchronization, UTF-16/file-URI conversion, dirty targets, stale requests/diagnostics, and transport failures. Stack checks cover independent initialization, diagnostic merging/clearing, sibling failure isolation in active/parked buffers, same-revision rename/close identity, aggregate limits, first-server position routing, and both shutdowns. Semantic ordering uses protocol/file gates; elapsed deadlines only bound stuck subprocesses.
- Hover regression checks cover content forms, sanitization, cancelled/superseded replies, invalid results, scrolling/dismissal, Unicode clipping, and preserved editing state.
- Semantic-token checks cover full/range requests, legend/modifier remapping, UTF-16 boundaries and grapheme rounding, refresh/change coalescing, stale rename/close replies, null/empty/error fallback, primary/sibling retirement, and rendered cell precedence with syntax, diagnostics, search, selections, and carets.
Clipboard checks use isolated providers rather than reading or overwriting the desktop clipboard.
+2
View File
@@ -175,3 +175,5 @@ Space begins a leader sequence in Normal/Visual by default. Press the suffix key
Buffers highlight automatically using [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) and compiled-in grammar/query packages. The selected language set is Rust, Ruby, Python, JSON, Bash, Makefile, Markdown, JavaScript/JSX, TypeScript/TSX, HTML, CSS, YAML, TOML, C, C++, and Go. Detection uses filename/extension or the in-memory first-line shebang, including common special filenames such as Makefile, Gemfile, and Cargo.lock; no file reread occurs. Unsupported formats use the normal foreground. The detected language appears in status, and save-as redetects it.
Colors retain the terminal palette: comments dark gray, strings green, keywords magenta, numbers yellow, types/markup cyan, and functions light blue. Grammar queries handle multiline/nested constructs, even when their opening is above the viewport. Markdown uses its inline grammar and named fenced-code injections; HTML script/style content uses JavaScript/CSS. Known tagged-template and raw-string injections use the same selected grammar registry; unknown injected languages retain host highlighting or plain text. Highlighting styles whole graphemes and expanded tab cells; partially clipped wide placeholders remain unchanged. Search, Visual selections, and secondary carets retain foreground/background precedence.
An enabled supporting LSP server supplies semantic tokens automatically, overriding syntax colors only on recognized spans. Variables/parameters use light cyan, properties blue, and readonly variables/properties yellow; declarations/definitions are bold, documentation comments italic, and deprecated symbols struck through. Tree-sitter fills gaps and remains available while new tokens are pending after edits. Search, Visual selections, diagnostics, and carets keep their existing precedence. See [native LSP configuration](configuration.md#native-lsp) for provider support and server-stack routing.