diff --git a/docs/architecture.md b/docs/architecture.md index 3c73fb9..aa495e2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -115,6 +115,8 @@ The same cancellable worker collects normal-picker status for the startup projec Each buffer has a frontend-owned parser/tree cache. Unchanged revision/path skips comparison and parsing; unchanged text at a later revision also avoids parsing. Actual text changes produce a minimal UTF-8-safe byte edit with newline-aware Tree-sitter Points, edit the prior tree, then parse using that tree for incremental reuse. Grammar changes reset the tree, including TypeScript versus TSX despite their shared display name. Native highlight/local/injection queries resolve capture precedence and scoped definitions into nonoverlapping original-line spans; unchanged injected trees are reused. Buffer switching restores its cache, and closing retires it. Incomplete text and grammar ERROR nodes remain highlightable; genuine language/query/parser failures expose a status error and readable plain text. Full-tree highlight queries still run after incremental parsing—a deliberate ceiling to revisit if measured costly. There is no local semantic inference engine or runtime grammar download. +Full-source syntax work is bounded to 1 MiB, with a maximum 64 KiB source line (UTF-8 bytes, including normalized line separators in the total). Oversized buffers retain language detection and an explicit plain-text status, but allocate no source copy, per-row spans, scopes, or parse tree. Crossing either budget releases existing parser/cache allocations; unchanged oversized revisions return immediately, and shrinking below both budgets resumes normal parsing. The same policy covers ordinary buffers, diff sources including their separately stored final newline, and text-only hover caches. + Diff highlighting lazily owns two instances of the same syntax cache, keyed by the accepted comparison request ID and selected file. Before detects language from the old path when renamed; After uses the new path. Scrolling, focus changes, and explorer toggling reuse parsed spans without copying/reparsing snapshots. A fresh comparison identity prevents stale Working-checkout colors even when labels and paths are unchanged. Closing the view releases snapshot caches. Syntax styles patch foreground/modifiers over diff backgrounds; historical panes never receive live-buffer semantic tokens. Scope context is a lazily built row index over the existing primary/injected syntax trees, invalidated by source or grammar changes and warmed by diff updates. Ordinary buffer highlighting does not build it. The frontend's visual-row projection inserts inline/sticky scope labels in wrapped and unwrapped diffs; source alignment and excerpt ranges contain no label rows. Coloring, native selection/caret overlays, and source navigation skip these presentation rows. Before/After labels use their independent trees rather than live-workspace symbol analysis. diff --git a/docs/development.md b/docs/development.md index a14ce1b..e34fc8d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -92,7 +92,10 @@ Diff styling checks cover independent snapshot language/shebang/injection detect ## Performance and process limits -- Undo uses full-buffer snapshots and movement scans the current line. This favors straightforward correctness over large-file performance. +- Retained undo/redo history stores only changed rows. Insert/replay/completion transactions still use temporary full-buffer snapshots, and undo/redo reconstructs the buffer. Ordinary single-line edits with one pane and no semantic overlay modify the line in place; multiline, multi-edit, shared-pane, and semantic-overlay batches retain the general whole-buffer path. Dirty-state comparisons are cached until text or the saved baseline changes. Movement and wrapping still scan the current line. +- Syntax falls back to plain text above 1 MiB of source or a 64 KiB line, before source copies, tree construction, or per-row syntax allocations. The detected language and fallback reason remain visible; reducing the buffer below both budgets restores highlighting. These are optional syntax limits, not file-opening or editing limits. +- Saves use buffered writes before flushing, syncing, and atomically replacing the destination. External-change checks remain independent of dirty-state caching. Large-file verification must include exact saved bytes and edit/undo/redo round-trips, not startup alone. +- Blame and gutter snapshots are size-checked before joining buffer lines; files above their existing 16 MiB limits retain editing but have no annotations. Eligibility is cached with the source revision and recovers after shrinking. - Rapid-wheel verification must replay bursts against the running editor and measure arrival at the final viewport after input stops, not just individual frame duration. Cheap frames can still accumulate a visible backlog if each queued report requires its own redraw. - Keyboard viewport scrolling coalesces up to 64 consecutive inputs for the same resolved scroll action into one redraw. Reversals, changes between row/configured-step scrolling, and cursor motions end the batch so boundary clamping and caret reveal happen before the next action. Burst checks must include direction changes at the start/end of the buffer, not just throughput. - PTY throughput alone cannot prove terminal presentation stays smooth: capture the editor and terminal while the user reproduces a pause. Synchronized-output commands must not reach the terminal during frame computation, even when individual frames and queued-input catch-up appear fast. diff --git a/docs/editing.md b/docs/editing.md index 4d61ac5..3cc3832 100644 --- a/docs/editing.md +++ b/docs/editing.md @@ -590,6 +590,8 @@ config.bind "leader f", config.core.follow_link Buffers highlight automatically using [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) and compiled-in grammar/query packages. The selected language set is Rust, Ruby, Python, JSON, Bash, Makefile, Markdown, JavaScript/JSX, TypeScript/TSX, HTML, CSS, YAML, TOML, C, C++, Go, Zig (`.zig`), and Honey (`.hon`). Detection uses filename/extension or the in-memory first-line shebang, including common special filenames such as Makefile, Gemfile, and Cargo.lock; no file reread occurs. Unsupported formats use the normal foreground. The detected language appears in status, and save-as redetects it. +Files remain editable above the syntax budget: sources over 1 MiB, or with any line over 64 KiB, use plain text and show the reason in the language status. Syntax highlighting resumes automatically after shrinking below both limits. This avoids building a full parse tree for huge generated files; navigation, search, selections, undo/redo, and saving remain available. + Colors retain the terminal palette: comments dark gray, strings green, keywords magenta, numbers yellow, types/markup cyan, and functions light blue. Grammar queries handle multiline/nested constructs, even when their opening is above the viewport. Markdown uses its inline grammar and named fenced-code injections; HTML script/style content uses JavaScript/CSS. Known tagged-template and raw-string injections use the same selected grammar registry; unknown injected languages retain host highlighting or plain text. Highlighting styles whole graphemes and expanded tab cells; partially clipped wide placeholders remain unchanged. Search, Visual selections, and secondary carets retain foreground/background precedence. Honey uses its bundled grammar and highlight queries, including declarations, primitive types, literals, and comments. Markdown fences accept `honey`, `honeylang`, or `hon`; the language ID is `honey`. diff --git a/src/blame.rs b/src/blame.rs index 4e7e98d..f6f622e 100644 --- a/src/blame.rs +++ b/src/blame.rs @@ -13,7 +13,7 @@ use std::{ time::{Duration, Instant}, }; -const LIMIT: usize = 16 * 1024 * 1024; +pub(crate) const LIMIT: usize = 16 * 1024 * 1024; const TIMEOUT: Duration = Duration::from_secs(5); static NEXT_TEMP: AtomicU64 = AtomicU64::new(0); type ResultFile = Result>>>, String>; diff --git a/src/editor.rs b/src/editor.rs index 22786c7..6f6f074 100644 --- a/src/editor.rs +++ b/src/editor.rs @@ -295,6 +295,40 @@ struct Snapshot { primary: usize, } +/// Retain only changed rows between undo groups, not another copy of the file. +#[cfg_attr(test, derive(Debug))] +struct HistoryEntry { + start: usize, + end: usize, + before: Snapshot, +} + +impl HistoryEntry { + fn new(mut before: Snapshot, after: &[String]) -> Self { + let start = before + .lines + .iter() + .zip(after) + .take_while(|(a, b)| a == b) + .count(); + let suffix = before.lines[start..] + .iter() + .rev() + .zip(after[start..].iter().rev()) + .take_while(|(a, b)| a == b) + .count(); + before.lines = before + .lines + .drain(start..before.lines.len() - suffix) + .collect(); + Self { + start, + end: after.len() - suffix, + before, + } + } +} + struct Edit<'a> { target: Option, start: usize, @@ -1868,6 +1902,8 @@ struct Buffer { snapshot_root: Option, lines: Vec, saved_lines: Vec, + // Invalidate whenever text or the saved baseline changes, including temporary swaps. + dirty_cache: Cell>, disk_stamp: Option, reload_error: Option, text_revision: u64, @@ -1878,9 +1914,8 @@ struct Buffer { view: ViewState, path: Option, identity: Option, - // ponytail: full-buffer history costs O(buffer size) per action; use edit deltas for large files. - undo: Vec, - redo: Vec, + undo: Vec, + redo: Vec, } impl Buffer { @@ -1926,6 +1961,7 @@ impl Buffer { snapshot_document_path: None, snapshot_root: None, saved_lines: lines.clone(), + dirty_cache: Cell::new(Some(false)), disk_stamp, reload_error: None, lines, @@ -1946,7 +1982,15 @@ impl Buffer { } fn dirty(&self) -> bool { - !self.resolver_readonly && self.diff_snapshot.is_none() && self.lines != self.saved_lines + if self.resolver_readonly || self.diff_snapshot.is_some() { + return false; + } + if let Some(dirty) = self.dirty_cache.get() { + return dirty; + } + let dirty = self.lines != self.saved_lines; + self.dirty_cache.set(Some(dirty)); + dirty } fn refresh_file(&mut self) -> io::Result { @@ -1994,6 +2038,7 @@ impl Buffer { } self.replace_lines(split_lines(&text)); self.saved_lines.clone_from(&self.lines); + self.dirty_cache.set(Some(false)); self.undo.clear(); self.redo.clear(); self.view.reload(&self.lines); @@ -2029,12 +2074,17 @@ impl Buffer { if let Some(tokens) = &mut self.semantic_tokens { retain_semantic_tokens(tokens, &self.lines, &lines); } - self.text_revision = self.text_revision.wrapping_add(1); - self.diagnostics.clear(); - self.view.search_match = None; + self.text_changed(); } std::mem::replace(&mut self.lines, lines) } + + fn text_changed(&mut self) { + self.text_revision = self.text_revision.wrapping_add(1); + self.dirty_cache.set(None); + self.diagnostics.clear(); + self.view.search_match = None; + } } #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -2601,15 +2651,18 @@ impl Editor { let seed = split_lines(projection.seed()); if buffer.lines != seed { let previous = buffer.replace_lines(seed); - buffer.undo.push(Snapshot { - lines: previous, - targets: if buffer.view.targets.is_empty() { - vec![Target::caret(Cursor::default())] - } else { - buffer.view.targets.clone() + buffer.undo.push(HistoryEntry::new( + Snapshot { + lines: previous, + targets: if buffer.view.targets.is_empty() { + vec![Target::caret(Cursor::default())] + } else { + buffer.view.targets.clone() + }, + primary: buffer.view.primary, }, - primary: buffer.view.primary, - }); + &buffer.lines, + )); buffer.redo.clear(); buffer.view.clamp(&buffer.lines); for pane in self.panes.values_mut() { @@ -2675,12 +2728,14 @@ impl Editor { buffer.view.clamp(&buffer.lines); } buffer.saved_lines = saved; + buffer.dirty_cache.set(None); buffer.identity = Some(target_path.clone()); buffer.path = Some(target_path); buffer.resolver_readonly = false; } else if !buffer.dirty() { buffer.replace_lines(vec!["Non-text resolution saved; reopen the file to inspect its current type.".into()]); buffer.saved_lines.clone_from(&buffer.lines); + buffer.dirty_cache.set(Some(false)); buffer.path = None; buffer.identity = None; buffer.resolver_readonly = true; @@ -2983,6 +3038,7 @@ impl Editor { if buffer.lines != alternative { buffer.replace_lines(alternative.clone()); buffer.saved_lines = alternative; + buffer.dirty_cache.set(Some(false)); buffer.view.clamp(&buffer.lines); for pane in self.panes.values_mut() { if let Some(view) = pane.views.get_mut(&alternative_buffer) { @@ -5024,6 +5080,7 @@ impl Editor { buffer.snapshot_document_path = Some(document_path); buffer.snapshot_root = Some(root); buffer.lines = lines; + buffer.dirty_cache.set(None); buffer } @@ -10028,6 +10085,7 @@ impl Editor { match atomic_save(&path, &self.active.lines, expected) { Ok(stamp) => { self.active.saved_lines.clone_from(&self.active.lines); + self.active.dirty_cache.set(Some(false)); self.active.save_revision = self.active.save_revision.wrapping_add(1); self.active.disk_stamp = Some(stamp); self.active.reload_error = None; @@ -10063,7 +10121,9 @@ impl Editor { fn record(&mut self, before: Snapshot) { if before.lines != self.active.lines { self.active.view.search_match = None; - self.active.undo.push(before); + self.active + .undo + .push(HistoryEntry::new(before, &self.active.lines)); self.active.redo.clear(); } } @@ -11369,6 +11429,7 @@ impl Editor { let linewise = self.active.view.linewise; let search_match = self.active.view.search_match; let revision = self.active.text_revision; + let dirty_cache = self.active.dirty_cache.get(); let diagnostics = std::mem::take(&mut self.active.diagnostics); let semantic_tokens = self.active.semantic_tokens.clone(); let mut restored = false; @@ -11380,6 +11441,7 @@ impl Editor { // Validate the entire replay first, then publish while the original // document is restored, and only then commit the computed deletion. std::mem::swap(&mut self.active.lines, &mut before.lines); + self.active.dirty_cache.set(None); std::mem::swap(&mut self.active.view.targets, &mut before.targets); std::mem::swap(&mut self.active.view.primary, &mut before.primary); restored = true; @@ -11389,6 +11451,7 @@ impl Editor { shape, )?; std::mem::swap(&mut self.active.lines, &mut before.lines); + self.active.dirty_cache.set(None); std::mem::swap(&mut self.active.view.targets, &mut before.targets); std::mem::swap(&mut self.active.view.primary, &mut before.primary); restored = false; @@ -11412,6 +11475,7 @@ impl Editor { self.active.view.primary = before.primary; } self.active.text_revision = revision; + self.active.dirty_cache.set(dirty_cache); self.active.diagnostics = diagnostics; self.active.semantic_tokens = semantic_tokens; self.active.view.mode = mode; @@ -11677,13 +11741,35 @@ impl Editor { self.normalize_targets(); } + fn restore_history(&mut self, entry: HistoryEntry) -> HistoryEntry { + let mut lines = self.active.lines.clone(); + let end = entry.start + entry.before.lines.len(); + let before = Snapshot { + lines: lines + .splice(entry.start..entry.end, entry.before.lines) + .collect(), + targets: self.active.view.targets.clone(), + primary: self.active.view.primary, + }; + self.restore(Snapshot { + lines, + targets: entry.before.targets, + primary: entry.before.primary, + }); + HistoryEntry { + start: entry.start, + end, + before, + } + } + fn undo(&mut self) { if self.mutation_blocked() { return; } if let Some(before) = self.active.undo.pop() { - self.active.redo.push(self.snapshot()); - self.restore(before); + let reverse = self.restore_history(before); + self.active.redo.push(reverse); } else { self.set_message("Nothing to undo"); } @@ -11694,8 +11780,8 @@ impl Editor { return; } if let Some(after) = self.active.redo.pop() { - self.active.undo.push(self.snapshot()); - self.restore(after); + let reverse = self.restore_history(after); + self.active.undo.push(reverse); } else { self.set_message("Nothing to redo"); } @@ -11951,6 +12037,61 @@ impl Editor { if edits.is_empty() { return; } + // Ordinary typing/deletion can edit one line without copying the entire file. + // Snapshot previews, shared panes and semantic overlays use the general remapping path. + if original.is_none() + && edits.len() == 1 + && self.panes.len() == 1 + && self.active.semantic_tokens.is_none() + && !edits[0].text.contains('\n') + { + let edit = &edits[0]; + let mut offsets = line_offsets(&self.active.lines); + let row = offsets + .partition_point(|offset| *offset <= edit.start) + .saturating_sub(1); + let start = edit.start - offsets[row]; + let end = edit.end - offsets[row]; + if end <= self.active.lines[row].len() { + let mut destinations: Vec<_> = self + .active + .view + .targets + .iter() + .map(|target| { + ( + translate_edit_offset( + cursor_offset(&self.active.lines, &offsets, target.active), + true, + std::iter::once(edit), + ), + true, + ) + }) + .collect(); + if let Some(target) = edit.target { + destinations[target] = (edit.start + edit.caret, edit.round_up); + } + if self.active.lines[row][start..end] != *edit.text { + self.active.lines[row].replace_range(start..end, &edit.text); + self.active.text_changed(); + } + for offset in &mut offsets[row + 1..] { + *offset = *offset - (end - start) + edit.text.len(); + } + for (target, (byte, round_up)) in + self.active.view.targets.iter_mut().zip(destinations) + { + target.active = offset_cursor(&self.active.lines, &offsets, byte, round_up); + target.anchor = target.active; + target.preferred_columns = None; + } + if !preserve_targets { + self.normalize_targets(); + } + return; + } + } edits.sort_by_key(|edit| (edit.start, edit.end)); let mut unique: Vec = Vec::with_capacity(edits.len()); for (index, edit) in edits.iter().enumerate() { @@ -13115,7 +13256,7 @@ fn atomic_save( .parent() .filter(|parent| !parent.as_os_str().is_empty()) .unwrap_or(Path::new(".")); - let (temporary, mut file) = loop { + let (temporary, file) = loop { let id = TEMP_ID.fetch_add(1, Ordering::Relaxed); let temporary = parent.join(format!(".vamp-{}-{id}.tmp", std::process::id())); let mut options = OpenOptions::new(); @@ -13129,12 +13270,15 @@ fn atomic_save( } }; let result = (|| { + let mut writer = io::BufWriter::new(&file); for (index, line) in lines.iter().enumerate() { if index > 0 { - file.write_all(b"\n")?; + writer.write_all(b"\n")?; } - file.write_all(line.as_bytes())?; + writer.write_all(line.as_bytes())?; } + writer.flush()?; + drop(writer); if let Some(metadata) = metadata { file.set_permissions(metadata.permissions())?; } diff --git a/src/editor_review.rs b/src/editor_review.rs index e133897..d6ac720 100644 --- a/src/editor_review.rs +++ b/src/editor_review.rs @@ -109,6 +109,7 @@ impl Editor { .is_some_and(|binding| binding.owner == self.tab_id) { buffer.saved_lines.clone_from(&buffer.lines); + buffer.dirty_cache.set(Some(false)); } } } @@ -268,6 +269,7 @@ impl Editor { }) { buffer.review_buffer = None; buffer.saved_lines.clone_from(&buffer.lines); + buffer.dirty_cache.set(Some(false)); } } self.autosave_local_review()?; @@ -849,6 +851,7 @@ impl Editor { self.active.lines = split_lines(text); self.active.saved_lines.clone_from(&self.active.lines); self.active.text_revision = self.active.text_revision.wrapping_add(1); + self.active.dirty_cache.set(Some(false)); self.normalize_targets(); } } else { diff --git a/src/gutter.rs b/src/gutter.rs index 247e0b8..8d26f7b 100644 --- a/src/gutter.rs +++ b/src/gutter.rs @@ -19,7 +19,7 @@ use std::{ thread::{self, JoinHandle}, }; -const FILE_LIMIT: usize = 16 * 1024 * 1024; +pub(crate) const FILE_LIMIT: usize = 16 * 1024 * 1024; const BATCH_LIMIT: usize = 64 * 1024 * 1024; const FILE_COUNT_LIMIT: usize = 1024; type Files = Vec<(usize, PathBuf, Arc)>; diff --git a/src/session.rs b/src/session.rs index 84211d5..27d07e5 100644 --- a/src/session.rs +++ b/src/session.rs @@ -425,7 +425,18 @@ struct GutterSource { save_revision: u64, path: PathBuf, dirty: bool, - contents: Arc, + contents: Option>, +} + +fn text_fits(lines: &[String], limit: usize) -> bool { + lines + .iter() + .try_fold(lines.len().saturating_sub(1), |bytes, line| { + bytes + .checked_add(line.len()) + .filter(|bytes| *bytes <= limit) + }) + .is_some() } /// Owns command eligibility and execution hosts together. Frontends can inspect @@ -1479,8 +1490,11 @@ impl EditorSession { source.revision == document.revision && source.path == path }) .map_or_else( - || Arc::from(document.lines.join("\n")), - |source| Arc::clone(&source.contents), + || { + text_fits(document.lines, crate::gutter::FILE_LIMIT) + .then(|| Arc::from(document.lines.join("\n"))) + }, + |source| source.contents.clone(), ); Some(GutterSource { buffer: document.id, @@ -1492,6 +1506,16 @@ impl EditorSession { }) }) .collect(); + for source in &self.gutter_sources { + if source.contents.is_none() { + self.editor.set_gutter_changes( + source.buffer, + source.revision, + &source.path, + Vec::new(), + ); + } + } } self.gutter_request_id = self.gutter_request_id.wrapping_add(1); self.last_gutter_refresh = Some(now); @@ -1508,12 +1532,12 @@ impl EditorSession { .gutter_sources .iter() .filter(|_| line_gutter) - .map(|source| { - ( + .filter_map(|source| { + Some(( source.buffer, source.path.clone(), - Arc::clone(&source.contents), - ) + Arc::clone(source.contents.as_ref()?), + )) }) .collect(); let picker = self.picker_source.as_ref().map(|root| { @@ -1521,7 +1545,9 @@ impl EditorSession { .gutter_sources .iter() .filter(|source| source.dirty) - .map(|source| (source.path.clone(), Arc::clone(&source.contents))) + .filter_map(|source| { + Some((source.path.clone(), Arc::clone(source.contents.as_ref()?))) + }) .collect(); (root.clone(), buffers) }); @@ -1624,7 +1650,9 @@ impl EditorSession { self.blame_request_id = self.blame_request_id.wrapping_add(1); self.blame_lines = None; changed |= self.editor.clear_line_blame(); - if let Some(source) = &self.blame_source { + if let Some(source) = &self.blame_source + && text_fits(self.editor.lines(), crate::blame::LIMIT) + { if self.blame_worker.is_none() { self.blame_worker = BlameWorker::start().ok(); } diff --git a/src/syntax.rs b/src/syntax.rs index 6b2ec2a..a19555d 100644 --- a/src/syntax.rs +++ b/src/syntax.rs @@ -14,6 +14,12 @@ pub(crate) const RUST_ID: &str = "rust"; pub(crate) const RUBY_ID: &str = "ruby"; pub(crate) const PYTHON_ID: &str = "python"; +// Full trees and their indexes are optional; editing remains available beyond these budgets. +const MAX_SYNTAX_BYTES: usize = 1024 * 1024; +const MAX_SYNTAX_LINE_BYTES: usize = 64 * 1024; +const SOURCE_BUDGET_NOTICE: &str = "source exceeds 1 MiB syntax budget"; +const LINE_BUDGET_NOTICE: &str = "line exceeds 64 KiB syntax budget"; + pub(crate) fn is_markdown_path(path: &Path) -> bool { languages::detect(Some(path), None).is_some_and(|syntax| syntax.name == "Markdown") } @@ -69,6 +75,7 @@ pub struct SyntaxCache { path: Option, final_newline: bool, syntax: Option<&'static LanguageSpec>, + budget_exceeded: Option<&'static str>, source: String, next_source: String, line_starts: Vec, @@ -100,7 +107,34 @@ impl SyntaxCache { } pub(crate) fn update_text(&mut self, text: &str, path: &Path) { - if self.source == text + // Check lengths before equality, splitting into owned lines, or copying the source. + let budget_exceeded = if text.len() > MAX_SYNTAX_BYTES { + Some(SOURCE_BUDGET_NOTICE) + } else if text + .split('\n') + .any(|line| line.len() > MAX_SYNTAX_LINE_BYTES) + { + Some(LINE_BUDGET_NOTICE) + } else { + None + }; + if let Some(reason) = budget_exceeded { + let syntax = languages::detect(Some(path), None) + .or_else(|| languages::detect(None, Some(syntax_prefix(text)))); + let language_changed = !same_language(self.syntax, syntax); + self.revision = None; + self.final_newline = false; + if self.path.as_deref() != Some(path) { + self.path = Some(path.to_path_buf()); + } + self.syntax = syntax; + if self.budget_exceeded != Some(reason) || language_changed { + self.use_plain_text(reason); + } + return; + } + if self.budget_exceeded.is_none() + && self.source == text && self.path.as_deref() == Some(path) && self.highlighted.iter().all(|highlighted| *highlighted) { @@ -168,8 +202,11 @@ impl SyntaxCache { let path_changed = self.path.as_deref() != path; let revision_changed = self.revision != Some(revision) || self.final_newline != final_newline; + if self.budget_exceeded.is_some() && !path_changed && !revision_changed { + return; + } let syntax = if path_changed || revision_changed { - languages::detect(path, lines.first().map(String::as_str)) + languages::detect(path, lines.first().map(|line| syntax_prefix(line))) } else { self.syntax }; @@ -181,12 +218,29 @@ impl SyntaxCache { } self.syntax = syntax; + let length = if revision_changed { + syntax_source_length(lines, final_newline) + } else if let Some(reason) = self.budget_exceeded { + Err(reason) + } else { + Ok(self.source.len()) + }; + let length = match length { + Ok(length) => length, + Err(reason) => { + if self.budget_exceeded != Some(reason) || language_changed { + self.use_plain_text(reason); + } + return; + } + }; + if self.budget_exceeded.take().is_some() { + self.error = None; + } + let mut edit = None; if revision_changed { self.next_source.clear(); - let length = lines.iter().map(String::len).sum::() - + lines.len().saturating_sub(1) - + usize::from(final_newline); self.next_source.reserve(length); for (row, line) in lines.iter().enumerate() { if row > 0 { @@ -289,6 +343,27 @@ impl SyntaxCache { } } + fn use_plain_text(&mut self, reason: &'static str) { + // Drop parser arenas as well as source/span capacities when crossing the budget. + *self = Self { + revision: self.revision, + path: self.path.take(), + final_newline: self.final_newline, + syntax: self.syntax, + budget_exceeded: Some(reason), + error: self + .syntax + .map(|syntax| format!("{}: plain text ({reason})", syntax.name)), + #[cfg(test)] + parses: self.parses, + #[cfg(test)] + incremental_parses: self.incremental_parses, + #[cfg(test)] + injection_parses: self.injection_parses, + ..Self::default() + }; + } + /// Sorted, non-overlapping, end-exclusive ranges. Unclassified text uses the normal foreground. pub fn line(&self, row: usize) -> &[SyntaxSpan] { self.lines.get(row).map_or(&[], Vec::as_slice) @@ -352,7 +427,7 @@ impl SyntaxCache { self.syntax.map(|syntax| syntax.name) } - /// Genuine grammar/query failures disable highlighting; ERROR nodes in user text do not. + /// Why highlighting is unavailable: a deliberate plain-text fallback or grammar/query failure. pub fn error(&self) -> Option<&str> { self.error.as_deref() } @@ -546,6 +621,34 @@ impl SyntaxCache { } } +fn syntax_prefix(text: &str) -> &str { + let mut end = text.len().min(MAX_SYNTAX_LINE_BYTES); + while !text.is_char_boundary(end) { + end -= 1; + } + text[..end].split('\n').next().unwrap_or_default() +} + +fn syntax_source_length(lines: &[String], final_newline: bool) -> Result { + let mut length = lines + .len() + .saturating_sub(1) + .saturating_add(usize::from(final_newline)); + if length > MAX_SYNTAX_BYTES { + return Err(SOURCE_BUDGET_NOTICE); + } + for line in lines { + if line.len() > MAX_SYNTAX_LINE_BYTES { + return Err(LINE_BUDGET_NOTICE); + } + length = length.saturating_add(line.len()); + if length > MAX_SYNTAX_BYTES { + return Err(SOURCE_BUDGET_NOTICE); + } + } + Ok(length) +} + fn has_scope_language(language: &str) -> bool { matches!( language, diff --git a/tests/unit/diff.rs b/tests/unit/diff.rs index cf5263a..3781314 100644 --- a/tests/unit/diff.rs +++ b/tests/unit/diff.rs @@ -1844,7 +1844,10 @@ fn column_view_preserves_split_gaps_unified_sides_and_folded_coordinates() { let ColumnRow::Text(line) = before.row(1).unwrap() else { panic!("unified insertion must paint the after source"); }; - assert_eq!((line.text, line.side, line.source_row), (Some("added"), Focus::After, 1)); + assert_eq!( + (line.text, line.side, line.source_row), + (Some("added"), Focus::After, 1) + ); assert_eq!(line.numbers, [None, Some(1)]); assert!(line.changed); assert_eq!(before.native_row(1), None); diff --git a/tests/unit/editor.rs b/tests/unit/editor.rs index 8a7cb56..83f8f27 100644 --- a/tests/unit/editor.rs +++ b/tests/unit/editor.rs @@ -323,6 +323,7 @@ fn resolver_explorer_retains_last_jj_conflict_state_until_confirmation() { ResolverStatus::Unconfirmed ); editor.active.saved_lines.clone_from(&editor.active.lines); + editor.active.dirty_cache.set(None); editor.refresh_resolver_status(); assert_eq!( editor.resolver().unwrap().files[0].status, @@ -3245,7 +3246,10 @@ fn review_identity_changes_retire_drafts_before_reusing_comment_ids() { let restored = root.join("restored.json"); editor.execute(&format!("diff.review.save {}", restored.display())); assert_eq!( - crate::review::Review::load(&restored, &root).unwrap().comments[0].body, + crate::review::Review::load(&restored, &root) + .unwrap() + .comments[0] + .body, "Original feedback" ); @@ -7145,19 +7149,25 @@ mod system_clipboard { editor .apply(vec![Effect::Replace("aa\nbb\ncc".into())]) .unwrap(); + editor.active.saved_lines.clone_from(&editor.active.lines); + editor.active.dirty_cache.set(Some(false)); + assert!(!editor.dirty()); type_text(&mut editor, "Vj"); fixture.fail("fail-write", true); let before = state(&editor); editor.handle(Input::Char('.')); assert_eq!(state(&editor), before); + assert!(!editor.dirty()); assert_eq!(fixture.bytes("state"), b"a\n"); assert!(editor.visual_linewise()); fixture.fail("fail-write", false); editor.handle(Input::Char('.')); assert_eq!(editor.context().text, "cc"); assert_eq!(fixture.bytes("state"), b"aa\nbb\n"); + assert!(editor.dirty()); editor.handle(Input::Char('u')); assert_eq!(editor.context().text, "aa\nbb\ncc"); + assert!(!editor.dirty()); assert_eq!(editor.mode(), Mode::Normal); } @@ -7795,11 +7805,12 @@ fn filter_escape_resume_and_clear_do_not_edit_underlying_buffers_or_history() { editor.count = Some(7); let snapshot = |state: &Snapshot| (state.lines.clone(), state.targets.clone(), state.primary); + let history = |entry: &HistoryEntry| (entry.start, entry.end, snapshot(&entry.before)); let lines = editor.lines().to_owned(); let targets = editor.active.view.targets.clone(); let primary = editor.active.view.primary; - let undo: Vec<_> = editor.active.undo.iter().map(&snapshot).collect(); - let redo: Vec<_> = editor.active.redo.iter().map(&snapshot).collect(); + let undo: Vec<_> = editor.active.undo.iter().map(&history).collect(); + let redo: Vec<_> = editor.active.redo.iter().map(&history).collect(); let insert = editor.active.view.insert_start.as_ref().map(snapshot); let buffer = editor.buffer_id(); let buffers = editor.buffer_count(); @@ -7845,11 +7856,11 @@ fn filter_escape_resume_and_clear_do_not_edit_underlying_buffers_or_history() { assert_eq!(editor.buffer_id(), buffer); assert_eq!(editor.buffer_count(), buffers); assert_eq!( - editor.active.undo.iter().map(&snapshot).collect::>(), + editor.active.undo.iter().map(&history).collect::>(), undo ); assert_eq!( - editor.active.redo.iter().map(&snapshot).collect::>(), + editor.active.redo.iter().map(&history).collect::>(), redo ); assert_eq!( @@ -8078,11 +8089,12 @@ fn selection_navigation_wraps_without_touching_any_underlying_mode() { assert!(len > 1); let snapshot = |state: &Snapshot| (state.lines.clone(), state.targets.clone(), state.primary); + let history = |entry: &HistoryEntry| (entry.start, entry.end, snapshot(&entry.before)); let text = editor.context().text; let targets = editor.active.view.targets.clone(); let primary = editor.active.view.primary; - let undo: Vec<_> = editor.active.undo.iter().map(&snapshot).collect(); - let redo: Vec<_> = editor.active.redo.iter().map(&snapshot).collect(); + let undo: Vec<_> = editor.active.undo.iter().map(&history).collect(); + let redo: Vec<_> = editor.active.redo.iter().map(&history).collect(); let insert = editor.active.view.insert_start.as_ref().map(snapshot); editor.count = Some(7); for (input, selected) in [ @@ -8104,11 +8116,11 @@ fn selection_navigation_wraps_without_touching_any_underlying_mode() { assert_eq!(editor.active.view.command_line, "original prompt"); assert_eq!(editor.count(), Some(7)); assert_eq!( - editor.active.undo.iter().map(&snapshot).collect::>(), + editor.active.undo.iter().map(&history).collect::>(), undo ); assert_eq!( - editor.active.redo.iter().map(&snapshot).collect::>(), + editor.active.redo.iter().map(&history).collect::>(), redo ); assert_eq!( @@ -12182,6 +12194,7 @@ fn palette_cancellation_preserves_every_underlying_mode_and_insert_group() { } let snapshot = |state: &Snapshot| (state.lines.clone(), state.targets.clone(), state.primary); + let history = |entry: &HistoryEntry| (entry.start, entry.end, snapshot(&entry.before)); let lines = editor.active.lines.clone(); let targets = editor.active.view.targets.clone(); let primary = editor.active.view.primary; @@ -12190,8 +12203,8 @@ fn palette_cancellation_preserves_every_underlying_mode_and_insert_group() { let search = editor.active.view.search_query.clone(); let found = editor.active.view.search_match; let message = editor.message.clone(); - let undo: Vec<_> = editor.active.undo.iter().map(&snapshot).collect(); - let redo: Vec<_> = editor.active.redo.iter().map(&snapshot).collect(); + let undo: Vec<_> = editor.active.undo.iter().map(&history).collect(); + let redo: Vec<_> = editor.active.redo.iter().map(&history).collect(); let insert = editor.active.view.insert_start.as_ref().map(snapshot); editor.handle(Input::Ctrl('p')); @@ -12218,11 +12231,11 @@ fn palette_cancellation_preserves_every_underlying_mode_and_insert_group() { assert_eq!(editor.active.view.search_match, found); assert_eq!(editor.message, message); assert_eq!( - editor.active.undo.iter().map(&snapshot).collect::>(), + editor.active.undo.iter().map(&history).collect::>(), undo ); assert_eq!( - editor.active.redo.iter().map(&snapshot).collect::>(), + editor.active.redo.iter().map(&history).collect::>(), redo ); assert_eq!( @@ -13489,6 +13502,56 @@ fn insert_sessions_and_open_lines_are_single_undo_actions() { assert_eq!(editor.context().text, "hello\n世界"); } +#[test] +fn large_buffer_history_retains_changed_rows_and_round_trips_all_groups() { + let directory = Directory::new(); + let path = directory.path("large.txt"); + let source = format!("head\n{}tail\n", "unchanged contents\n".repeat(10_000)); + fs::write(&path, &source).unwrap(); + let mut editor = Editor::open(Some(path), vec![]).unwrap(); + for _ in 0..12 { + editor.handle(Input::Char('i')); + editor.handle(Input::Char('界')); + editor.handle(Input::Escape); + } + assert_eq!(editor.lines()[0], format!("{}head", "界".repeat(12))); + let retained_bytes: usize = editor + .active + .undo + .iter() + .flat_map(|entry| &entry.before.lines) + .map(String::len) + .sum(); + assert!( + retained_bytes < 4096, + "small edits retained {retained_bytes} bytes of history" + ); + for _ in 0..12 { + editor.handle(Input::Char('u')); + } + assert_eq!(editor.lines().join("\n"), source); + assert!(!editor.dirty()); + for _ in 0..12 { + editor.handle(Input::Char('U')); + } + assert_eq!(editor.lines()[0], format!("{}head", "界".repeat(12))); + assert_eq!(&editor.lines()[1..], &split_lines(&source)[1..]); + assert!(editor.dirty()); + editor.handle(Input::Char('u')); + editor.handle(Input::Char('i')); + editor.handle(Input::Enter); + editor.handle(Input::Escape); + assert!(editor.active.redo.is_empty()); + editor.handle(Input::Char('u')); + assert_eq!(editor.lines()[0], format!("{}head", "界".repeat(11))); + editor.execute("w"); + assert!(!editor.dirty()); + assert_eq!( + fs::read_to_string(editor.path().unwrap()).unwrap(), + format!("{}{source}", "界".repeat(11)) + ); +} + #[test] fn grapheme_navigation_and_deletion_do_not_split_unicode() { let mut editor = editor(); @@ -14076,6 +14139,52 @@ fn saves_preserve_empty_files_lf_trailing_newlines_and_dirty_history() { assert!(editor.should_quit()); } +#[test] +fn dirty_cache_tracks_saved_text_across_edits_undo_restore_and_reload() { + let directory = Directory::new(); + let path = directory.path("file.txt"); + fs::write(&path, "saved\n").unwrap(); + let mut editor = Editor::open(Some(path.clone()), vec![]).unwrap(); + editor.set_clipboard_mode("separate").unwrap(); + assert!(!editor.dirty()); + + editor + .apply(vec![Effect::Insert("changed".into())]) + .unwrap(); + assert!(editor.dirty()); + for _ in 0..3 { + assert!(editor.dirty()); + assert!(editor.active.info(true).dirty); + assert!(editor.active.document().dirty); + assert!(!editor.active.refresh_file().unwrap()); + } + editor.handle(Input::Char('u')); + assert!(!editor.dirty()); + editor.handle(Input::Char('U')); + assert!(editor.dirty()); + + editor + .apply(vec![Effect::Replace("saved\n".into())]) + .unwrap(); + assert!(!editor.dirty()); + editor.apply(vec![Effect::Insert("new ".into())]).unwrap(); + assert!(editor.dirty()); + let revision = editor.text_revision(); + editor.execute("w"); + assert_eq!(editor.text_revision(), revision); + assert!(!editor.dirty()); + editor.handle(Input::Char('u')); + assert!(editor.dirty()); + editor.handle(Input::Char('U')); + assert!(!editor.dirty()); + + fs::write(&path, "reloaded content\n").unwrap(); + assert!(editor.active.refresh_file().unwrap()); + assert_eq!(editor.lines(), ["reloaded content", ""]); + assert!(!editor.dirty()); + assert!(!editor.active.refresh_file().unwrap()); +} + #[test] fn save_as_and_failed_wq_keep_path_dirty_state_and_quit_correct() { let directory = Directory::new(); diff --git a/tests/unit/session.rs b/tests/unit/session.rs index ce8164f..e2ac2ce 100644 --- a/tests/unit/session.rs +++ b/tests/unit/session.rs @@ -4,6 +4,47 @@ use std::{ sync::atomic::{AtomicU64, Ordering}, }; +#[test] +fn oversized_repository_text_skips_snapshots_and_recovers_after_shrinking() { + let fixture = Fixture::new(); + let path = fixture.file("buffer.txt", "small\n"); + let mut session = EditorSession::open(Some(path.clone()), vec![], None).unwrap(); + let buffer = session.editor.buffer_id(); + session.editor.set_gutter_changes( + buffer, + session.editor.text_revision(), + &path, + vec![Some(crate::gutter::Change::Added)], + ); + assert_eq!( + session.editor.gutter_change(0), + Some(crate::gutter::Change::Added) + ); + fs::write(&path, "x".repeat(crate::gutter::FILE_LIMIT + 1)).unwrap(); + assert!(session.editor.refresh_files()); + session.poll_gutter_at(Instant::now()); + session.synchronize_documents(); + let source = session + .gutter_sources + .iter() + .find(|source| source.buffer == buffer) + .unwrap(); + assert!(source.contents.is_none()); + assert!(session.blame_worker.is_none()); + assert_eq!(session.editor.gutter_change(0), None); + fs::write(&path, "small again\n").unwrap(); + assert!(session.editor.refresh_files()); + session.poll_gutter_at(Instant::now()); + session.synchronize_documents(); + let source = session + .gutter_sources + .iter() + .find(|source| source.buffer == buffer) + .unwrap(); + assert_eq!(source.contents.as_deref(), Some("small again\n")); + assert!(session.blame_worker.is_some()); +} + #[test] fn resolver_load_errors_return_through_real_session_without_touching_the_draft() { let fixture = Fixture::new(); diff --git a/tests/unit/syntax.rs b/tests/unit/syntax.rs index f7e91d2..78b3eae 100644 --- a/tests/unit/syntax.rs +++ b/tests/unit/syntax.rs @@ -729,6 +729,152 @@ fn unchanged_revision_and_same_source_do_not_allocate_or_parse() { assert_eq!(cache.incremental_parses, 0); } +fn assert_plain_text_storage(cache: &SyntaxCache) { + assert!(cache.tree.is_none()); + assert!(cache.context.is_none()); + assert!(cache.scopes.get().is_none()); + assert_eq!(cache.source.capacity(), 0); + assert_eq!(cache.next_source.capacity(), 0); + assert_eq!(cache.line_starts.capacity(), 0); + assert_eq!(cache.lines.capacity(), 0); + assert_eq!(cache.string_spans.capacity(), 0); + assert_eq!(cache.highlighted.capacity(), 0); + assert_eq!(cache.injections.capacity(), 0); + assert_eq!(cache.captures.capacity(), 0); + assert_eq!(cache.highlights.capacity(), 0); + assert_eq!(cache.events.capacity(), 0); + assert!(cache.line(0).is_empty()); + assert!(cache.string_spans(0).is_empty()); + assert!(cache.scope_at(0).is_none()); +} + +#[test] +fn syntax_budget_crossing_releases_state_and_reenables_after_shrinking() { + let small = "fn main() { let value = \"ok\"; }"; + let mut input = lines(small); + let mut remaining = MAX_SYNTAX_BYTES - small.len(); + while remaining > 0 { + let length = (remaining - 1).min(16 * 1024); + input.push(" ".repeat(length)); + remaining -= length + 1; + } + let path = Some(Path::new("budget.rs")); + let mut cache = SyntaxCache::new(); + cache.update_visible(&input, path, 0, std::slice::from_ref(&(0..1))); + assert_eq!(cache.source.len(), MAX_SYNTAX_BYTES); + assert_eq!(cache.error(), None); + assert_eq!(cache.parses, 1); + assert_token(&cache, small, 0, "fn", SyntaxKind::Keyword); + assert!(cache.scope_at(0).is_some()); + input.last_mut().unwrap().push(' '); + cache.update_visible(&input, path, 1, std::slice::from_ref(&(0..1))); + assert_eq!(cache.language(), Some("Rust")); + assert!(cache.error().unwrap().contains("plain text")); + assert_eq!(cache.parses, 1); + assert_plain_text_storage(&cache); + let notice = cache.error.as_ref().unwrap().as_ptr(); + cache.update(&input, path, 1); + assert_eq!(cache.error.as_ref().unwrap().as_ptr(), notice); + assert_eq!(cache.parses, 1); + assert_plain_text_storage(&cache); + cache.update(&input, Some(Path::new("budget.py")), 1); + assert_eq!(cache.language(), Some("Python")); + assert!(cache.error().unwrap().starts_with("Python")); + assert_plain_text_storage(&cache); + cache.update(&input, Some(Path::new("budget.txt")), 1); + assert_eq!(cache.language(), None); + assert_eq!(cache.error(), None); + assert_plain_text_storage(&cache); + cache.update(&lines(small), path, 2); + assert_eq!(cache.parses, 2); + assert_eq!(cache.error(), None); + assert_token(&cache, small, 0, "fn", SyntaxKind::Keyword); + assert!(cache.scope_at(0).is_some()); +} + +#[test] +fn diff_syntax_budget_counts_the_separate_final_newline() { + let mut input = vec!["//".to_owned() + &" ".repeat(16 * 1024 - 2); 63]; + input.push("//".to_owned() + &" ".repeat(16 * 1024 - 65)); + let path = Some(Path::new("budget.rs")); + let mut cache = SyntaxCache::new(); + cache.update_diff(&input, path, 0, false, None); + assert_eq!(cache.source.len(), MAX_SYNTAX_BYTES); + assert_eq!(cache.parses, 1); + assert_eq!(kind_at(&cache, 0, 0), Some(SyntaxKind::Comment)); + cache.update_diff(&input, path, 0, true, None); + assert_eq!(cache.parses, 1); + assert_plain_text_storage(&cache); + cache.update_diff(&input, path, 0, false, Some(std::slice::from_ref(&(0..1)))); + assert_eq!(cache.parses, 2); + assert_eq!(cache.error(), None); + assert_eq!(kind_at(&cache, 0, 0), Some(SyntaxKind::Comment)); +} + +#[test] +fn long_lines_and_large_text_never_copy_or_parse_and_clear_injections() { + let markdown = "# Title\n```rust\nfn main() { \"ok\"; }\n```"; + let mut cache = self::cache(markdown, "hover.md"); + assert!(!cache.injections.is_empty()); + assert!(cache.scope_at(2).is_some()); + let parses = cache.parse_counts(); + let long_line = "//".to_owned() + &" ".repeat(MAX_SYNTAX_LINE_BYTES - 1); + cache.update_text(&long_line, Path::new("large.rs")); + assert_eq!(cache.language(), Some("Rust")); + assert_eq!(cache.parse_counts(), parses); + assert!(cache.error().unwrap().contains("line")); + assert_plain_text_storage(&cache); + let large = " ".repeat(52 * 1024 * 1024); + cache.update_text(&large, Path::new("parser.c")); + assert_eq!(cache.language(), Some("C")); + assert_eq!(cache.parse_counts(), parses); + assert_plain_text_storage(&cache); + let notice = cache.error.as_ref().unwrap().as_ptr(); + cache.update_text(&large, Path::new("parser.c")); + assert_eq!(cache.error.as_ref().unwrap().as_ptr(), notice); + assert_plain_text_storage(&cache); + let large_lines = vec![large]; + cache.update(&large_lines, Some(Path::new("parser.c")), 10); + assert_eq!(cache.language(), Some("C")); + assert_eq!(cache.parse_counts(), parses); + assert_plain_text_storage(&cache); + cache.update_diff(&large_lines, Some(Path::new("parser.c")), 10, true, None); + assert_eq!(cache.parse_counts(), parses); + assert_plain_text_storage(&cache); + cache.update_text(markdown, Path::new("hover.md")); + assert!(cache.parse_counts().0 > parses.0); + assert_eq!(cache.error(), None); + assert!(!cache.injections.is_empty()); + assert!(cache.scope_at(2).is_some()); + let boundary = "//".to_owned() + &" ".repeat(MAX_SYNTAX_LINE_BYTES - 2); + cache.update(&lines(&boundary), Some(Path::new("line.rs")), 50); + assert_eq!(cache.error(), None); + assert_eq!(kind_at(&cache, 0, 0), Some(SyntaxKind::Comment)); + cache.update(&lines(&long_line), Some(Path::new("line.rs")), 51); + assert_plain_text_storage(&cache); + cache.update_text("", Path::new("line.rs")); + assert_eq!(cache.language(), Some("Rust")); + assert_eq!(cache.error(), None); + assert!(cache.budget_exceeded.is_none()); + assert!(cache.line(0).is_empty()); + assert!(cache.scope_at(0).is_none()); + + let mut script = vec!["#!/usr/bin/env python3".to_owned(), long_line.clone()]; + cache.update(&script, None, 60); + assert_eq!(cache.language(), Some("Python")); + assert_plain_text_storage(&cache); + script[0] = "#!/usr/bin/env ruby".to_owned(); + cache.update(&script, None, 61); + assert_eq!(cache.language(), Some("Ruby")); + assert!(cache.error().unwrap().starts_with("Ruby")); + assert_plain_text_storage(&cache); + script.pop(); + cache.update(&script, None, 62); + assert_eq!(cache.language(), Some("Ruby")); + assert_eq!(cache.error(), None); + assert!(cache.tree.is_some()); +} + #[test] fn unicode_minimal_edits_use_byte_points_and_never_split_codepoints() { let old = "x\n\t\"é 👩‍💻\"\n";