honey/TODO.md

# self-hosting

- [ ] milestone: Honey compiles its production compiler, then that compiler compiles itself again and builds/runs the example programs.

## current boundary

directory packages support file-local import namespaces and postfix declaration
selection through the interpreter and native backend. concrete 8/16/32/64-bit
integers, checked numeric conversions, records/tuples, construction/defaults,
enums, distinct/alias types, overlapping untagged/tagged unions, field reads/writes
and exhaustive statement/value matching work through both execution paths.
source data pointers/references support aliasing, mutation, calls and aggregate
storage; explicit callback signatures, many-pointer addition, mutable/sentinel
slices, optionals and layout/cast/memory builtins work through both execution paths.
native scalar/pointer `c_func` declarations support allocator-required libc calls;
executed foreign calls remain diagnostic traps in the interpreter.
explicit fallible channels, composed error sets, `try`/`catch` and scoped LIFO
`defer`/`errdefer` run through both paths; source errors are values, not diagnostic
traps or compiler infrastructure failures.
cached generic specialization and type-producing procedures support explicit/inferred
compile-time arguments and generated record/array storage through both paths.
ordinary procedures can compute scalar, nested record/array, tagged-union and
fallible compile-time values using mutable locals, loops and recursive calls;
explicit `$()` expressions use the same constant evaluation path. compile-time
`if`/`match` selection and type comparisons specialize only the selected body,
including owned by-value tagged payload captures in ordinary generic procedures.
type-producing procedures also accept ordinary
constant inputs, cached and erased like explicit compile-time arguments. mutable
type/value locals, direct and aggregate field/index assignments, branches,
scalar/enum/tagged-union matching and bounded integer-range/collection/while loops now generate types and storage. optional
if/while captures and tagged payload matches select nested constructors, lengths, signatures and defaults;
array/slice/string/pointer-to-array collection loops support exact copy/pointer
captures and `usize` indices through both paths; type-producing bodies iterate
materializable arrays/strings by value without retaining interpreter addresses.
type-valued fixed arrays support nested copies, indexed/whole updates,
checked projections and by-value collection captures in type-producing bodies;
`undefined` type/value arrays track initialization per leaf while loops fill generated types/defaults.
type-array and type-record parameters/results cross compile-time procedure boundaries
with owned snapshots, cached nominal identity and erased arguments.
initialized type-valued records/tuples support schemas/defaults, nested
field/index updates and owned copies/captures inside compile-time producers.
`field!` selects record/tuple/enum and sequence members using compile-time names,
including runtime places and type-control updates without runtime reflection.
`struct_type!` generates ordinary/C-layout records from constant name/type/default
arrays or tuples, including loop-filled producer storage and owned defaults.
`inline for` expands constant ranges, fixed value/type arrays, heterogeneous
value/type tuples and strings with immutable exact captures and compile-time-selected exits.
final inline match arms expand nominal enums and tagged unions with static tags
and exact by-value payloads, preserving runtime dispatch and scoped control flow.
general type-valued execution, metadata reflection and generic library acceptance remain open.
the driver retains frontend storage and source provenance through
`@source/compilation`, with compilation-wide AST node identities.
file commands retain single-file behavior; directory commands compile the
package's production files and load its dependencies in separate scopes.

## completed milestone: scalar-function packages

verified: compile and run a small multi-file directory package importing a second
package of scalar functions, through both the interpreter and native backend.
using the existing standard library is a later acceptance gate: even `std/mem`
needs records, pointers, generics and C calls.

- [x] retain source/package identity and source ownership across the compilation; keep one identifier pool and unambiguous declaration IDs.
- [x] load directory packages once per canonical path, with deterministic file discovery, package scopes and root-only entry selection. follow Bro's existing relative and project-root `@` import rules; diagnose cycles rather than recurse indefinitely.
- [x] finish postfix namespace selection and resolve file-local import aliases without injecting dependency names into the importing package; bare and aliased import declarations are already parsed for dependency discovery.
- [x] reuse ordinary scalar signature checking, function values and IR calls for selected declarations; namespaces have no runtime representation.
- [x] preserve originating filenames and spans in dependency diagnostics and native runtime traps; retain demand-driven diagnostic recovery.
- [x] verify sibling source files, transitive/shared imports, same-spelled declarations in different packages, alias conflicts, missing packages/members, argument errors and dependency-file diagnostics. preserve existing single-file commands.

visibility follows Bro's public-by-default contract; unsupported visibility
attributes must be diagnosed, never ignored. the scalar-function milestone did not
include record fields or imported types; those now work for the record/tuple slice.
test discovery and FFI remain outside these completed slices.

## subsequent dependency gates

- [x] concrete data models: required integer widths/conversions, records/tuples and construction, fields, enums, distinct/alias types, unions and matching. examples: `source/lexer/token.hon`, `source/sema/resolve.hon`, `std/mem/mem.hon`.
  - [x] native integer widths (`i8`–`i64`, `u8`–`u64`, `isize`/`usize`) and checked explicit numeric conversions, including float truncation/range checks and integer-to-float rounding. preserve exact types through arrays, calls and range captures; retain strict implicit conversion rules.
  - [x] records/tuples, typed/contextual/anonymous construction, field defaults and reads/writes, nested aggregate value copies, imported record types and constructors through interpreter/native execution.
  - [x] nominal enums and distinct types, transparent declaration/type aliases, overlapping untagged/tagged unions, contextual payload construction, active-field checks and exhaustive scalar/enum/tagged-union matching through interpreter/native execution. pointer captures, inline specialization and yielding value blocks remain with their respective later gates; error-set composition is covered below.
- [x] memory representation: source pointers/references, mutable and sentinel slices, optionals, layout/cast builtins and allocator-required libc calls. internal IR array addresses are not source pointers. examples: `std/mem/mem.hon`, `std/arraylist/arraylist.hon`.
  - [x] source data pointers/references: `@T`/`*T`, `mut`, address-taking and dereference, pointer-aware fields/indexing, array `.len`/`.ptr`, readonly weakening, location equality, recursive pointer-linked records and shared pointees across interpreter/native calls and aggregate copies.
  - [x] function-pointer signatures: explicit `@proc(...) Result`, structural signature checking, callbacks in parameters/results/records/arrays, vtable calls and imported signature aliases; many-pointer `+ usize` arithmetic preserves pointee type, mutability, stride and location identity through interpreter/native execution.
  - [x] mutable/sentinel slices with checked bounds, borrowed aliases and string views; optional values/pointers/callbacks, null checks, unwrap and conditional/loop captures; target layout, pointer/const casts and typed memory copy/set through interpreter/native execution. scalar/pointer C ABI declarations build/run allocator callbacks using `malloc`, `realloc`, `free` and `posix_memalign`; arbitrary interpreter FFI, generic standard-library compilation and native startup/IO remain with later gates.
- [x] fallible results and cleanup: explicit `T ! E`, error-set composition, `try`/`catch`, `defer`/`errdefer`, error captures and matched error residuals through interpreter/native execution; infrastructure errors remain distinct from diagnostic traps. fallible `void` may complete on fallthrough; unhandled entry errors exit with status `1`. inferred `T!` signatures and yielding catch blocks remain with inference/value-control-flow work. examples: `source/main.hon`, `source/sema/check/check.hon`.
- [ ] generic specialization and compile-time execution/reflection sufficient for existing `ArrayList`, hash maps, `enummap`, `meta` and `io.print`; include type/value parameters and compile-time generated storage.
  - [x] specialization checkpoint: explicit/inferred type and integer/boolean/string/callback/null value parameters, cached concrete runtime calls, nominal type-producing procedures, generated record/array storage and recursive data pointers through interpreter/native execution. preserve declared value types, source diagnostics and repeatable compilation; runtime values cannot supply compile-time arguments.
  - [x] compile-time execution checkpoint: ordinary procedure calls with constant inputs and integer/boolean/string results reuse checking, typed IR and bounded interpreter execution. mutable scalar locals, range/while loops, break/continue, nested/generic/recursive calls, exact numeric types, runtime traps and dependency-file provenance survive argument erasure through interpreter/native execution. runtime parameters and mutable global dependencies are rejected; recompilation retains no prior execution state.
  - [x] aggregate/explicit-expression checkpoint: computed nested records/tuples and fixed arrays, contextual/dependent aggregate value parameters, normalized specialization caching and explicit `$()` expressions through interpreter/native execution. preserve by-value isolation, zero-cell shapes, integer widths, string ownership, enum/optional fields and exact IEEE float bits; reject runtime dependencies and values that would retain temporary interpreter storage. repeatable compilation restores erased expressions.
  - [x] conditional specialization checkpoint: explicit compile-time `if` guards and side-effect-free constant guards in concrete generic procedures select a body before specialization/checking. type equality retains aliases, nominal identity and structural signatures; composed error-set identity/cache keys ignore ordering, grouping and repetition. guard checking, selected-body diagnostics, scoped cleanup, runtime guards and repeatable compilation work through interpreter/native execution; both arms remain parsed and name-resolved.
  - [x] constant-input type-procedure checkpoint: type-producing procedures accept ordinary parameters alongside `$` parameters, require every argument explicitly and evaluate/cache/erase all inputs. generated record/array storage, nominal cache identity, nested calls, dependent contextual aggregate inputs, exact erased-argument checking, source diagnostics, imported producers and repeatable compilation work through interpreter/native execution.
  - [x] mutable type-execution checkpoint: type-producing bodies execute mutable type/value locals, direct binding assignments, scoped blocks, if/else and early returns, integer-range/while loops, break/continue and nested/recursive producers. generated nested types, scalar-computed lengths/defaults, nominal cache identity, exact value types, overflow/guard/assignment diagnostics, imported producers and repeatable compilation work through interpreter/native execution. type-control execution shares a 4096-step bound across nested producers; scalar operations reuse checked IR execution. aggregate field/index updates and type-valued arrays are covered below; cleanup and type-valued record storage remain open.
  - [x] type-match checkpoint: type-producing bodies execute scalar/enum match statements and direct type-valued match expressions in initializers, assignments and returns, including nested selected arms. ordinary checking/IR execution select the arm and validate subject/pattern types, ranges, overlap, unreachable arms and exhaustiveness; only the selected body is elaborated. generated storage, scoped locals, loop transfers, early returns, nominal cache identity, selected-body diagnostics, dependency provenance and repeatable compilation work through interpreter/native execution. all bodies remain parsed/name-resolved and execution shares the type-control bound.
  - [x] nested type-match checkpoint: matches select types inside constructors, array lengths, generic arguments, local annotations, callback signatures and field defaults in type-producing bodies. selection precedes ordinary expansion, preserves literal spelling/source provenance and loop snapshots, and caches nominal results before expanding recursive fields. selected-body and pattern diagnostics, failed deferred elaboration, imported producers and repeatable compilation work through interpreter/native execution; deferred selection shares the producer's type-control bound.
  - [x] tagged/fallible constant checkpoint: explicit `$()` expressions, generic value parameters and type-procedure inputs materialize tagged unions and both fallible channels through interpreter/native execution. preserve active tags/payloads, nested raw aggregate projections, string ownership, void/zero-cell variants, composed/payload errors, declared channel types, normalized cache identity and repeatable compilation. `catch` computes generated lengths and runtime values. untagged unions and storage-bearing active payloads remain escape diagnostics rather than retaining temporary interpreter handles.
  - [x] tagged type-match checkpoint: type-producing bodies execute tagged-union matches with immutable by-value payload captures in statement, expression and nested storage positions. ordinary match checking validates every arm's captures, including grouped variants; selected payloads retain exact scalar/aggregate types and owned strings. generated storage/defaults, loop snapshots/transfers, cache identity, selected-body diagnostics, imported producers, dependency provenance and repeatable compilation work through interpreter/native execution. storage-bearing payloads retain the constant-materialization escape boundary.
  - [x] generic match-specialization checkpoint: side-effect-free constant scalar/enum/tagged-union subjects in concrete generic procedures and explicit `match $(subject)` expressions/statements select an arm before body specialization/checking. reuse ordinary checked IR for pattern types, ranges, overlap, exhaustiveness and every arm's capture validation; only the selected body is elaborated. exact by-value payload types/owned strings, runtime parameters, nested expression matches, cleanup, return/loop transfers, imported producers and repeatable compilation work through interpreter/native execution. ordinary call/runtime/mutable subjects stay runtime; contextual aggregates with possible runtime defaults are not implicitly evaluated. all bodies remain parsed/name-resolved; storage-bearing payloads retain the constant-materialization escape boundary.
  - [x] optional type-control checkpoint: type-producing bodies execute optional if/while guards with immutable by-value payload captures. ordinary checked IR validates guards; selected scalar/aggregate payloads retain exact types and owned strings across nested captures, generated storage/defaults, loop updates/transfers and early returns. absent optionals skip unselected bodies; invalid guards, capture mutation/shadowing, overflow and execution bounds retain diagnostics. imported producers, dependency provenance, nominal cache identity and repeatable compilation work through interpreter/native execution. null data/function pointers materialize correctly; nonnull storage-bearing payloads retain the constant-materialization escape boundary.
  - [x] aggregate type-update checkpoint: type-producing bodies update initialized, materializable aggregate locals through nested record fields and mutable fixed-array indices. ordinary checking validates assignments, retaining exact value types, mutability (including immutable payload captures), index types/bounds, unknown-field and narrow-overflow diagnostics. new owned constants preserve earlier by-value copies while later loops, generated storage and defaults observe updates. imported producers, dependency provenance, nominal cache identity and repeatable compilation work through interpreter/native execution. pointer/slice-backed mutation and partial record construction remain open; type-valued records/arrays and uninitialized value arrays are covered below.
  - [x] collection-iteration checkpoint: arrays, slices, strings and single pointers-to-arrays support immutable exact-type copy captures, storage-qualified pointer captures and optional `usize` indices through interpreter/native execution. iterables/bounds evaluate once, array lvalues remain borrowed and later element updates are visible; temporary arrays, aggregate copy isolation, zero-cell/empty collections, nested loops and cleanup across transfers retain ordinary semantics. ordinary constant-input procedures reuse the same checked IR; type-producing bodies iterate materializable arrays/strings with by-value captures, fixed entry-time place indices, generated storage, nominal cache identity, dependency diagnostics, shared execution bounds and repeatable compilation. storage-bearing type-control captures retain their existing boundary; type-valued elements are covered below.
  - [x] type-array storage checkpoint: type-producing bodies own initialized fixed arrays of type values, including explicit/inferred nested arrays, `.len`, checked `usize` indexing, whole/indexed updates and immutable by-value collection captures. persistent snapshots preserve copies and exact shape/mutability while borrowed array places observe later updates. selected types generate nested storage, constructors, signatures and defaults through interpreter/native execution, retaining selected-body and index/type/mutability diagnostics, imported provenance, nominal cache identity, shared bounds and repeatable compilation. uninitialized type arrays and local type-valued records are covered below; reflection remains open.
  - [x] type-array procedure checkpoint: fixed arrays of type values cross compile-time procedure inputs/results, including nested/aliased results, dependent `$N` lengths, empty shapes and erased `$types` parameters in runtime generic procedures. all inputs to type/type-array producers are explicit constants; owned snapshots preserve by-value isolation, exact shape/mutability and canonical/nominal cache identity. projections generate signatures, constructors, storage and defaults through interpreter/native execution. malformed inputs/results, runtime dependencies, imported provenance, shared execution bounds and repeatable compilation retain diagnostics and isolation. runtime type-array storage, type-valued record procedure boundaries and reflection remain open.
  - [x] field-selection reflection checkpoint: `field!(base, name)` evaluates/erases a constant string name and reuses ordinary selection through interpreter/native execution. records/tuples, enum/error-set members, tagged payloads, callbacks, pointer-qualified places and sequence `.len`/`.ptr` retain exact types, single base evaluation, address-taking, mutability and active-field checks. type-producing bodies snapshot changing selector names for nested aggregate updates without losing by-value isolation. runtime/nonstring names, malformed calls, missing members and noncanonical/overflowing tuple ordinals retain diagnostics; full metadata reflection and inline expansion remain open.
  - [x] uninitialized type-array checkpoint: fixed type arrays in type/type-array producers start with `undefined` and fill through loop-driven indexed, nested and whole-array updates. persistent local copies retain independent initialization state; `.len` works before leaves initialize. premature leaf reads and incomplete procedure arguments/results diagnose read-before-initialization without caching invalid values. explicit nonnegative dimensions, empty shapes, ordinary mutability/shape checks, nominal identity and repeatable compilation work through interpreter/native execution. construction shares the type-control bound per expanded cell, including nested zero-leaf shapes. ordinary value-array construction and local type-valued records are covered below; runtime undefined storage and metadata reflection remain open.
  - [x] uninitialized value-array checkpoint: ordinary fixed arrays in type/type-array producers start with `undefined`, fill nested indices or replace whole arrays/rows, and generate storage/defaults through interpreter/native execution. owned snapshots preserve independent partial initialization in local copies; initialized scalar/string/optional/record/tagged-union elements retain exact types and ordinary checked operations. premature leaf reads and incomplete procedure arguments diagnose read-before-initialization. empty shapes, `.len` before initialization, mutability/type/index/bounds/overflow diagnostics, per-cell construction limits and repeatable compilation are verified. runtime undefined storage, uninitialized arrays in ordinary value-returning procedures, partial record construction and storage-bearing constant materialization remain open.
  - [x] type-record storage checkpoint: initialized records/tuples containing type values live inside type/type-array producers, including local schemas/defaults, typed/contextual/inferred construction, nested records/arrays, checked field/index reads and updates, whole-value replacement and immutable by-value collection captures. persistent snapshots isolate copies/captures while borrowed array places observe later updates; type leaves retain nominal identity and ordinary value leaves reuse checked IR. generated storage/defaults, invalid field/type/default diagnostics, readonly places, narrow overflow, imported producers/provenance, cached nominal results and repeatable compilation work through interpreter/native execution. field validation/construction shares the type-control bound. type-record procedure inputs/results are covered below; runtime type-valued records, partial record initialization, metadata reflection and inline expansion remain open.
  - [x] type-record procedure checkpoint: records/tuples containing type values cross compile-time procedure inputs/results, including nested arrays/records, aliases, defaults and dependent annotations. compile-time producers accept ordinary constant inputs; runtime generics accept erased `$` metadata parameters. owned snapshots isolate argument/result copies, exact type/value leaves retain checked diagnostics, and normalized cache keys preserve nominal generated types. generated storage/defaults, imported producers/provenance, shared execution bounds and repeatable compilation work through interpreter/native execution. runtime metadata storage, partial record initialization, reflection and inline expansion remain open.
  - [x] generated-record checkpoint: `struct_type!(layout, names, types, defaults)` builds ordinary records from constant fixed arrays or tuples with unique ASCII field names, concrete types and contextual optional defaults. literal heterogeneous collections, helper results and loop-filled producer arrays preserve exact types, owned defaults/copies and cached nominal identity through interpreter/native execution. `.auto` permits empty records; `.c` requires nonempty C-compatible fields and retains native field order/layout. malformed inputs, incomplete arrays, runtime dependencies, field/default validation, imported provenance, shared construction bounds and repeatable compilation retain diagnostics and isolation. metadata inspection, inline expansion, aggregate C calling conventions and storage-bearing defaults remain open.
  - [x] inline-loop checkpoint: `inline for` expands constant integer ranges, fixed arrays (including type-valued arrays) and strings into separately checked runtime bodies through interpreter/native execution. exact immutable captures, `usize` collection indices, narrow inclusive maxima, selected conditional/tagged-match bodies, static break/continue tail pruning, nested inline/ordinary loops, runtime returns/errors and scoped cleanup retain ordinary semantics. empty loops skip body elaboration; runtime-dependent inputs/transfers, pointer captures, storage-bearing escapes and shared expansion bounds retain diagnostics. imported producers/provenance and repeatable compilation are verified. heterogeneous tuples and inline-match syntax are covered below; metadata iteration remains open.
  - [x] heterogeneous inline-tuple checkpoint: `inline for` iterates constant tuples with exact per-field value/type captures and `usize` indices through interpreter/native execution. literal/helper/aliased tuples, erased generic inputs, nested tuple projections, generated storage, empty shapes, static exits, cleanup and aggregate copy isolation retain existing semantics. type-producing bodies borrow tuple places so later fields observe updates while saved copies remain isolated. runtime dependencies and storage-bearing escapes are rejected before iteration, including discarded captures and early exits; named records and ordinary tuple loops remain invalid. imported producers, nominal type leaves and repeatable compilation are verified. inline-match syntax is covered below; metadata collections remain open.
  - [x] nominal inline-match checkpoint: final `inline |tag|:` enum arms and `inline |payload[, tag]|:` tagged-union arms expand remaining variants through interpreter/native execution. runtime subjects evaluate once; immutable static enum/tag captures specialize nested conditions, matches and generic calls while exact by-value scalar/aggregate/string/void payloads retain runtime semantics and copy isolation. statement/expression arms, ordinary loop/optional/match captures, scoped cleanup and return/break/continue work; type-producing bodies select generated storage and update outer type/value locals, including collection captures. invalid subjects/captures, redundant/nonfinal arms, ordinary pattern checks, imported provenance, shared expansion bounds and repeatable compilation retain diagnostics and ownership. composed error-set inline arms, pointer captures and metadata reflection remain open.
  - [ ] remaining gate: general type-valued compile-time execution, composed error-set inline arms and metadata reflection (`typeinfo!` and metadata collections), broader constant materialization (including untagged unions and storage-bearing values), plus existing-library acceptance. required expression-valued control flow, compound updates, formatting parsing and imported C-binding support may interleave with later gates.
- [ ] remaining source constructs: expression-valued control flow and `yield`, compound updates and multiline strings. examples: `source/parser/parse.hon`, `source/llvm/runtime.hon`.
- [ ] standard-library milestone: compile/run a multi-package program exercising existing allocation and IO, including error cleanup; no replacement miniature standard library.
- [ ] native driver integration: required C ABI declarations, `process.Init` startup arguments/IO, actual CLI exit statuses, file loading, temporary files and spawning Clang. Honey-generated programs currently accept only parameterless `main` and print its result; Bro's bootstrap startup support is not Honey backend support.
- [ ] self-hosting proof: bootstrap-built Honey builds the production compiler (excluding test-only imports); that executable rebuilds the compiler; the rebuilt compiler passes representative interpreter/native examples and CLI/error scenarios without Bro at runtime.

the gates may require interleaving (notably allocation, errors and generics).
extend checking, IR and native emission together; accepting syntax alone does
not complete a gate. additional targets, a package registry, a replacement
linker and interpreter execution of arbitrary FFI are not prerequisites.