feat: add asynchronous LSP plugin diagnostics and definitions

README.md

@@ -200,18 +200,19 @@ Leader bindings require Normal/Visual and the target's existing mode eligibility
 
 ## Module seams
 
-One crate, seven modules, no workspace or terminal-coordinate types in the editing backend:
+One crate, eight modules, no workspace or terminal-coordinate types in the editing backend:
 
 - `editor`: owns buffer lifecycle and per-buffer text, saved state, primary/selection set, modes, search, and undo/redo; session-wide clipboard policy, independent registers and owned-system fragment metadata, semantic native repeat, action metadata, command-palette and file-picker focus/dispatch, key notation, leader/count prefixes and binding resolution, modal paste policy, saving, command ownership, capabilities, and validated edit effects. Its low-level interface returns an `Invocation` when an external command is selected.
 - `files`: owns the gitignore-aware directory/file catalog, authoritative OS paths, compact tree projection, query filtering, browse/search focus, expansion/selection restoration, navigation, and scan/error state shared by floating and docked presentations.
 - `clipboard`: owns native plain-text provider selection and bounded subprocess I/O; it never computes selections or editing offsets.
 - `syntax`: owns Tree-sitter grammar detection, per-buffer incremental trees, bundled highlight/local/injection queries, and semantic byte spans; it has no terminal color/geometry or editing dependency.
-- `session`: `EditorSession` owns the editor and its Ruby hosts together. It loads configuration once, registers eligible plugins before installing shared bindings, executes selected commands against the active buffer, and retires both host and command ownership on invocation or effect-validation failure.
-- `tui`: translates terminal events to session input, renders read-only active-buffer state, keeps frontend-owned viewports and syntax caches by stable buffer ID, and advertises `text` and `styled_text` capabilities. It maps semantic tokens to terminal colors and has no plugin lifecycle callback, host storage, or paste acceptance policy.
+- `plugin`: owns generic document snapshots/events, zero-based grapheme positions, diagnostics, and versioned asynchronous update contracts. It has no LSP protocol types.
+- `session`: `EditorSession` owns the editor and its Ruby hosts together. It loads configuration once, registers eligible plugins before installing shared bindings, tracks document open/change/save/close events, polls asynchronous results, validates versioned diagnostics/navigation, and retires host and command ownership on invocation or effect-validation failure.
+- `tui`: translates terminal events to session input, renders read-only active-buffer state, keeps frontend-owned viewports and syntax caches by stable buffer ID, and advertises `text`, `styled_text`, and `diagnostics` capabilities. It maps semantic tokens and diagnostics to terminal styling and has no server/protocol logic or paste acceptance policy.
 - `ruby`: evaluates configuration and runs isolated plugin hosts. Protocol, deadlines, and process cleanup stay inside this module.
 - The executable selects paths and frontend capabilities, opens an editor session, and launches the TUI.
 
-Frontends drive `EditorSession::handle` or `execute` and inspect `editor()`; there is no mutable editor getter that could independently alter plugin registration. Pasted text uses `Input::Paste(String)` through the same input seam as keys. Plugin effects are applied internally so a rejected result also retires its owner; there is no generic frontend effects entry point.
+Frontends drive `EditorSession::handle` or `execute`, periodically call `poll()` even without input, and inspect `editor()`; there is no mutable editor getter that could independently alter plugin registration. `poll()` reports whether background activity requires redrawing. Pasted text uses `Input::Paste(String)` through the same input seam as keys. Plugin effects are applied internally so a rejected result also retires its owner; there is no generic frontend effects entry point.
 Frontends inspect `selections()` for document-ordered, end-exclusive Visual ranges of grapheme coordinates (empty outside Visual), `carets()` for all active cursors in document order, `cursor()` for the primary, and `selection_count()` for the total target count. A range ending at the next row's column 0 includes the previous row's newline; an empty final-row range contains no text. Selection normalization, clipboard contents, edit planning, and history remain inside the editor module.
 Frontends inspect `palette()` for the optional read-only query, filtered command entries, selected index, effective bindings, and eligibility. Palette input handling and acceptance stay in the editor; registered command execution stays in the session.
 Frontends use cached `selection_navigation_hint()` for command lists and `file_navigation_hint()` for the active file view/context, without reconstructing input policy.
@@ -219,6 +220,7 @@ Frontends inspect `count()` for the optional pending motion prefix; parsing and
 Frontends inspect `leader_pending()` for the cached display of a pending leader sequence; prefix recognition, cancellation, and dispatch remain editor-owned.
 Frontends inspect `buffer_id()`, `buffer_count()`, `has_buffer(id)`, and the ascending-ID `buffers()` iterator of borrowed path/dirty/active metadata. Existing text, cursor, selection, search, and mode methods describe only the active buffer.
 Frontends inspect `file_picker()` for query, presentation, view, explorer-local INSERT/filter subfocus, selection, and errors, and iterate borrowed `FileRow` metadata (actual path, display label, directory flag, visible depth, expansion). Dock side is a logical preference. They allocate rectangles, add folder suffixes/colors, and render; hierarchy, filtering, parent/child navigation, focus transitions, and selected-file opening remain backend-owned.
+Frontends inspect `diagnostics()` and `diagnostic_at_cursor()` for validated active-buffer annotations. Diagnostics are owned by a plugin and buffer revision; text changes and owner retirement clear them. The TUI underlines annotated text, colors affected gutter numbers by severity, and displays the primary caret's diagnostic message without replacing search/selection backgrounds.
 
 The terminal adapter owns raw mode, alternate-screen state, bracketed paste, and its panic hook for one session lifetime. Entry failure, normal return, I/O errors, and unwinding panics restore the caller's hook; active-terminal cleanup runs before caller panic diagnostics. Panics retain their original payload and propagate rather than being swallowed. Terminal state and panic hooks are process-global: run one TUI at a time and do not replace its hook while that session is active.
 
@@ -240,6 +242,28 @@ Built-in Ex commands register as the protected `core` feature in the same comman
 
 A feature declares all capabilities it needs. Registration is atomic: if any requirement is unavailable or any command conflicts, none of its commands are installed. For example, `Vamp.plugin "canvas", requires: ["graphics"]` will be refused by this TUI. Capability names are extensible strings, not a terminal-vs-GUI switch. Capability declarations express compatibility; they are not security permissions.
 
+### Optional LSP plugin
+
+LSP is implemented in `examples/plugins/lsp.rb`, not the editing core. It requires Ruby and an independently installed, trusted language server; it is not loaded by default. Enable it in your config (the plugin path is relative to that config):
+
+```ruby
+config.plugin "plugins/lsp.rb", options: {
+  servers: [
+    { language_id: "rust", extensions: ["rs"], command: ["rust-analyzer"] },
+    { language_id: "python", extensions: ["py"], command: ["pylsp"] }
+  ]
+}
+config.bind "leader d", "lsp_definition"
+```
+
+Server commands are argv arrays, never shell commands. The first matching extension configuration wins. One server runs per configuration/project root, using the startup working directory as its workspace. The plugin initializes the server, honors its open/change/save capabilities, synchronizes in-memory text (including hidden dirty buffers), and sends close events. Servers requesting incremental synchronization receive whole-document range replacements. Server transport and requests run in plugin worker threads; the editor continues processing input and idle diagnostics.
+
+`:lsp_definition` or the configured Space d binding requests a definition at the primary caret, once regardless of selection count. Both Location and LocationLink results are supported; multiple results use the first. Local-file navigation reuses loaded buffers without overwriting unsaved text. LSP UTF-16 positions and percent-encoded file URIs are translated to validated grapheme coordinates. Navigation is ignored if its source buffer, revision, caret, or latest request has changed.
+
+Diagnostics appear as severity-colored gutter numbers and text underlines; the primary caret's diagnostic appears in the message row. `:lsp_diagnostics` reports the current document's server state. Older versioned diagnostics are discarded; unversioned diagnostics can only be associated with the arrival-time document revision. Editing clears previous annotations immediately. Server startup/transport failures are reported and clear that server's diagnostics; other servers and native editing remain available. There are no automatic restarts.
+
+This plugin currently provides synchronization, diagnostics, and go-to-definition—not completion, hover, rename, formatting, code actions, workspace edits, or semantic highlighting. Tree-sitter remains the syntax highlighter. Server configuration requests return `null` for defaults; custom server settings and workspace discovery are not implemented.
+
 ## Ruby configuration
 
 By default, vamp loads `$XDG_CONFIG_HOME/vamp/config.rb`, or `~/.config/vamp/config.rb` when that environment variable is unset. `--config path` selects a file explicitly; `--no-config` bypasses Ruby entirely. Set `VAMP_RUBY` to select a different Ruby executable.
@@ -333,23 +357,33 @@ end
 
 Run `:greet vamp`, or press `g` with the example configuration. `examples/plugins/greeting.rb` also provides `:upper` to uppercase the buffer.
 
-Callbacks receive an immutable primary-only context snapshot: `editor.text`, `editor.row`, `editor.column`, `editor.mode`, and `editor.path`. Row/column are **one-based grapheme positions**; mode is `NORMAL`; path is a string or `nil`. Arguments are the remaining command text, without surrounding whitespace. Commands run once regardless of target count; successful text effects collapse to the primary before editing once. The plugin API does not expose selections.
+Callbacks receive an immutable primary-only context snapshot: `editor.text`, `editor.row`, `editor.column`, `editor.mode`, `editor.path`, `editor.buffer_id`, `editor.revision`, `editor.root`, and `editor.request_id`. Row/column are **one-based grapheme positions**; mode is `NORMAL`; path is a string or `nil`. Arguments are the remaining command text, without surrounding whitespace. Commands run once regardless of target count; successful text effects collapse to the primary before editing once. The plugin API does not expose selections.
 
 `editor.insert(text)`, `editor.replace(text)`, and `editor.message(text)` queue effects. The context remains the pre-command snapshot even after queuing an effect. All effects are validated before the backend changes anything; buffer changes from one successful invocation form one undo action. Exceptions, process exits, timeouts, malformed replies, and invalid effects disable that plugin and discard its effects. Other plugins and built-ins remain available. Plugin `puts`, explicit `STDOUT.write`, and stderr cannot overwrite the TUI; diagnostic output is discarded, while exceptions appear in the status message.
 
 This is crash isolation, **not a security sandbox**: Ruby configuration and plugins have your user account's filesystem/process permissions. Load only trusted scripts. On Unix, hosts run in their own process groups and their groups are killed on shutdown.
 
+### Asynchronous plugins
+
+`config.plugin "plugin.rb", options: { ... }` passes immutable JSON-compatible options as `Vamp.options`. Declare `Vamp.plugin "name", requires: ["text", "diagnostics"], async: true` to opt into asynchronous command results and document notifications. Existing plugins remain synchronous.
+
+`Vamp.on_document { |event| ... }` receives a deeply frozen hash with `"event"` (`"open"`, `"change"`, `"save"`, or `"close"`) and `"document"` (`"id"`, `"revision"`, `"path"`, `"root"`, `"text"`). Already-open documents are delivered at registration; changes include hidden-buffer edits, undo/redo, and plugin edits. Save-as sends close/open for the changed path. `Vamp.on_shutdown { ... }` supplies a shutdown hook. Callbacks must enqueue slow work rather than blocking host input.
+
+Worker threads call `Vamp.publish(effects, request_id: nil)`. Background effects are `message`, revision-tagged `diagnostics`, or request-correlated `navigate`; they cannot edit text. Diagnostics carry `buffer_id`, `revision`, and `items`, each with end-exclusive `start`/`end` positions, `severity` (`error`, `warning`, `information`, `hint`), and `message`. Positions are **zero-based grapheme coordinates**, not bytes, terminal cells, or UTF-16. Empty item lists clear the owner's annotations.
+
+Navigation carries the source `buffer_id`/`revision`, destination `path`/`row`/`column`, and the invocation's `editor.request_id` passed to `publish`. Only the latest still-current source request may move the primary. Closed/stale revisions are ignored; malformed current updates retire their owner. Failed navigation reports an error without discarding buffer edits. Hosts use bounded queues/frames; protocol failure or host exit retires commands, bindings, pending requests, and owned diagnostics.
+
 ### Messaging
 
-Rust and Ruby exchange UTF-8, newline-delimited JSON objects with `version: 1`. Startup returns `manifest` (name, required capabilities, commands) or `config` (plugin paths and bindings). `invoke` requests carry an ID, command, arguments, and context; `result` replies carry the matching ID and typed effects (`insert`, `replace`, `message`). Errors use `error` frames. Unknown fields, wrong versions, and mismatched replies are rejected. Frames and aggregate effect text are bounded to 4 MiB.
+Rust and Ruby exchange UTF-8, newline-delimited JSON objects with `version: 1`. Startup returns `manifest` (name, required capabilities, commands, async opt-in) or `config` (plugin paths/options and bindings). `invoke` requests carry an ID, command, arguments, and context; `result` replies carry the matching ID and typed text effects (`insert`, `replace`, `message`). Async hosts also receive `document`/`shutdown` and send `update` frames with optional request IDs and typed presentation/navigation effects. Errors use `error` frames. Unknown fields, wrong versions, and mismatched replies are rejected. Frames and aggregate effect text are bounded to 4 MiB.
 
 JSON uses Ruby's standard library. Protocol Buffers is not installed or required; there is no frontend/network wire protocol yet. The frontend currently calls the Rust library directly.
 
 ## Current limits
 
-- One visible buffer at a time; Normal, Insert, Command, Search, and characterwise Visual modes. No named registers, line/block selection, Vim operators, splits/tabs, regex/multiline search, LSP, or graphical frontend. Native system clipboard access currently supports macOS and Linux; Separate mode retains internal editing on unsupported/headless environments.
+- One visible buffer at a time; Normal, Insert, Command, Search, and characterwise Visual modes. No named registers, line/block selection, Vim operators, splits/tabs, regex/multiline search, or graphical frontend. Optional LSP currently supports diagnostics and definitions only. Native system clipboard access currently supports macOS and Linux; Separate mode retains internal editing on unsupported/headless environments.
 - Undo uses full-buffer snapshots and movement scans the current line. This favors straightforward correctness over large-file performance.
-- Plugin invocation is synchronous and may pause input for up to 500 ms before a host timeout. Config/plugin startup has a 2-second deadline per host. Buffer context must fit within a 4 MiB JSON request; a too-large request disables that plugin, not editing.
+- Legacy synchronous plugin invocation may pause input for up to 500 ms before a host timeout. Async plugins opt into background replies; pending requests and transport queues are bounded. Config/plugin startup has a 2-second deadline per host. Buffer context/document snapshots must fit within a 4 MiB JSON frame; a too-large frame disables that plugin, not editing.
 - Disabling a plugin removes its commands and bindings until the next launch. There is no hot reload.
 
 ## Verify
@@ -364,3 +398,5 @@ ruby -c src/ruby_host.rb
 Regression checks exercise Unicode edits, undo/redo, saving and failed saves, capability/command ownership, effect atomicity, real Ruby execution, protocol errors, crash/timeout recovery, and inherited-pipe cleanup. Backend paste checks cover ignored modes, literal Unicode/newline insertion, rejection atomicity, and typing/paste/typing undo and redo. Session checks join real Ruby execution to capability/collision rejection, complete command/binding retirement after invalid effects or host failure, surviving plugins, core editing/saving, and broken-config fallback. The executable was also exercised in PTY sessions for editing/saving, rejected-effect retirement, plugin crash and hang recovery, modal bracketed paste, rejected-paste feedback, undo/redo, scrolling, resize, and terminal restoration.
 
 Unix terminal-hook checks run in isolated child processes and real PTYs to keep global hooks out of the parallel test runner. They cover entry failure, normal return, I/O error, panic cleanup before caller diagnostics, original panic-payload propagation, and later panics producing no stale terminal-restoration output.
+
+LSP integration checks launch real framed-protocol fixture subprocesses through the optional Ruby plugin: fragmented/adjacent frames, initialization and synchronization capabilities, open/change/save/close, UTF-16 and encoded Unicode paths, dirty-buffer definitions, stale/superseded requests and diagnostics, unsupported server requests, crashes, malformed frames, unavailable servers, and surviving native editing. A real executable PTY smoke also exercised idle diagnostic redraw, active-server editing, definition navigation into a dirty loaded buffer, save/close notifications, and clean shutdown. No desktop clipboard or user files are touched.