feat: execute optional type-valued compile-time storage

Preserve owned optional metadata snapshots, nested presence, captures, checked unwraps and erased specialization identity. Verify imported nominal ownership across repeated compilation. Bootstrap: 299 passed. Native: 1080 programs plus 5 allocator programs passed with a temporary 120-second command deadline; bounded inline expansion exceeds the harness default of 30 seconds.

README.md

@@ -116,6 +116,8 @@ initialized records/tuples containing `type` fields live inside compile-time pro
 
 type-valued records/tuples may also start with `undefined`: `config struct { child type, count usize } := undefined`, then `config.child = i16` and `config.count = 2`. nested record fields, arrays of records, tuple ordinals and `field!` updates use the same persistent snapshots as initialized storage; partial copies retain independent initialization state. defaults do not initialize `undefined` fields. initialized leaves and array lengths can be read while other leaves remain uninitialized, but premature leaf reads and incomplete procedure arguments/results diagnose read-before-initialization. fully initialized results cross compile-time procedure boundaries and supply erased generic inputs, generated storage and defaults. empty shapes are initialized vacuously; each expanded record field/array cell shares the construction bound.
 
+optionals containing types also stay in owned compile-time storage: `?type`, nested `??type`, and optional records/arrays of metadata support null, contextual payloads, explicit `some!`, whole-value updates, immutable captures, null comparisons and checked unwraps. `some!(null)` preserves an absent inner optional inside a present outer one. constant-input procedures exchange these snapshots; erased generic inputs retain content-based specialization identity and nominal type leaves. `if`/`while` captures and conditional expressions select generated types or runtime bodies without runtime metadata. for generated array children, bind the unwrap first (`T :: selected?`, then `[count]T`); postfix `?` after an array type unwraps the whole expression. undefined presence and incomplete payloads diagnose before observation or procedure boundaries. runtime metadata storage and fallible type-valued channels remain unsupported.
+
 fixed type arrays may also start with `undefined` inside type/type-array producers: `types [2]mut type := undefined`, then fill `types[index] = i16` in a loop. nested and whole-array updates preserve shape, mutability and independent initialization state in local copies; `.len` is available before elements initialize. reading an uninitialized type leaf, returning an incomplete array or passing one to a compile-time procedure diagnoses a read before initialization. empty shapes are valid, but every dimension must be explicit and nonnegative. construction charges each expanded cell against the shared execution bound, including nested zero-leaf shapes.
 
 ordinary fixed value arrays use the same construction inside type/type-array producers: `counts [2]mut usize := undefined`, then fill indexed elements or replace whole rows/arrays. initialized leaves retain exact scalar, string, optional, record and tagged-union values; local copies retain independent partial initialization state. generated lengths/defaults and compile-time arguments can use initialized elements, while incomplete arrays cannot enter checked IR or procedure arguments. array `.len` and empty shapes do not read leaves. constant materialization of mutable slices or nonnull data pointers remains unsupported.

TODO.md

@@ -46,6 +46,8 @@ type-array and type-record parameters/results cross compile-time procedure bound
 with owned snapshots, cached nominal identity and erased arguments.
 initialized type-valued records/tuples support schemas/defaults, nested
 field/index updates and owned copies/captures inside compile-time producers.
+optional type-valued storage preserves nested presence, owned metadata payloads,
+checked unwraps and immutable captures across compile-time procedure boundaries.
 `field!` selects record/tuple/enum and sequence members using compile-time names,
 including runtime places and type-control updates without runtime reflection.
 `struct_type!` generates ordinary/C-layout records from constant name/type/default
@@ -155,6 +157,7 @@ test discovery and FFI remain outside these completed slices.
   - [x] untagged-union materialization checkpoint: address-free untagged unions own full native-layout byte snapshots through interpreter/native execution, including scalar/IEEE reinterpretation, nested padded records/arrays, smaller overlapping updates, zero-sized storage, distinct members and generated nominal unions. explicit `$()` values, compile-time arguments, full-byte specialization-cache identity, generated defaults, nested optionals/tagged payloads and both fallible channels preserve exact types and independent copies. uninitialized tail/padding bytes diagnose read-before-initialization; every member layout is checked recursively to reject data pointers, callbacks, strings and slices rather than retain interpreter handles. imported helpers/diagnostics, source/session disposal and repeated compilation retain ownership. address-bearing union layouts and broader constant materialization remain open.
   - [x] readonly typed-slice materialization checkpoint: ordinary compile-time calls snapshot readonly slices of materializable elements into module-lifetime backing through interpreter/native execution. nominal bytes, wider integers, arrays/records, source-bound callbacks, nested readonly slices, empty/zero-sized elements and full-width signed/unsigned sentinels retain exact types and logical lengths. explicit `$()` values, computed/direct content-based specialization identity, generated defaults and nested optional/tagged/fallible channels preserve ownership across helper returns, imported private declarations, source/session disposal and repeat compilation. initialization, address-range, sentinel, mutable-slice and nonnull-pointer escape diagnostics remain checked; 100,000-value/128-level snapshot bounds reject oversized and cyclic storage. mutable slices, nonnull data pointers and address-bearing untagged union layouts remain open.
   - [x] uninitialized type-record checkpoint: compile-time records/tuples containing type values start with `undefined` and fill through nested field/index updates or whole-value replacement. persistent snapshots preserve independent partial copies; defaults do not initialize undefined fields. initialized leaves and empty shapes work while premature type/value reads and incomplete procedure arguments/results diagnose read-before-initialization. fully initialized results supply erased generic inputs, generated storage/defaults and cached nominal types through interpreter/native execution. ordinary mutability/schema validation and shared per-field/cell construction bounds remain checked; general type-valued execution and broader materialization remain open.
+  - [x] optional type-storage checkpoint: `?type` and optional type-valued records/arrays use owned compile-time snapshots, including nested `some!(null)`, contextual payload construction, mutable whole-value/field/index replacement, immutable if/while/conditional captures, null comparisons and checked unwraps. constant-input procedures and erased generic arguments preserve specialization-cache identity and nominal leaves while generating runtime storage through interpreter/native execution. undefined presence, incomplete payloads, non-type values, capture mutation and runtime dependencies retain diagnostics. imported private nominal types, session/source disposal and repeat compilation preserve ownership. fallible type-valued channels, general type-valued execution and broader constant materialization remain open.
   - [ ] remaining gate: general type-valued compile-time execution, broader constant materialization (including address-bearing untagged unions, nonnull data pointers and mutable slices), plus remaining existing-library acceptance for broader `meta` operations and `io.print`. formatting parsing and broader imported C-binding support may interleave with later gates.
 - [ ] remaining source constructs: multiline strings. boolean and optional-capture conditional expressions, yielding value blocks, `yield`, compound updates and bitwise operators are covered above. examples: `source/parser/parse.hon`, `source/llvm/runtime.hon`.
 - [ ] standard-library milestone: compile/run a multi-package program exercising existing allocation and IO, including error cleanup; no replacement miniature standard library.

source/compilation/compilation.test.hon

@@ -1148,3 +1148,63 @@ computed_untagged_unions_recompile_without_borrowing_storage_bytes test {
         try testing.expect_equal(i64(42), outcome.value.data.integer)
     }
 }
+
+optional_type_metadata_recompiles_owned_private_nominal_payloads test {
+    diagnostic_store := diagnostics.init(mem.c_allocator)
+    defer diagnostics.deinit(&diagnostic_store)
+    modules [2]mut ir.Module := undefined
+    {
+        session := init(mem.c_allocator)
+        defer deinit(&session)
+        root :: try add_package(&session)
+        dependency :: try add_package(&session)
+        bytes := try owned_text(
+            `lib :: import "../library"
+            `Child :: struct { value bool }
+            `main proc() i16 {
+            `    if (lib.Box(lib.make()) != lib.Box(lib.make())) return -1
+            `    if (lib.Selected() == Child) return -2
+            `    value lib.Selected() :: { value = 42 }
+            `    return value.value
+            `}
+            ,
+        )
+        defer arraylist.deinit(&bytes)
+        root_source :: try add_source(&session, root, "/optional_types/app/main.hon", &bytes, &diagnostic_store)
+        imported := try owned_text(
+            `@hide:file
+            `Child :: struct { value i16 }
+            `@hide:file
+            `Config :: struct { child ?type, count usize }
+            `make proc() ?Config { return { child = Child, count = 2 } }
+            `Box proc(input ?Config) type {
+            `    info :: input?
+            `    T :: info.child?
+            `    return struct { values [info.count]T }
+            `}
+            `Selected proc() type {
+            `    if make() |info| { return info.child? }
+            `    return bool
+            `}
+            ,
+        )
+        defer arraylist.deinit(&imported)
+        _ = try add_source(&session, dependency, "/optional_types/library/types.hon", &imported, &diagnostic_store)
+        try arraylist.append(&session.imports, Import{
+            node = declaration(&session, root_source, 0).node,
+            target = dependency,
+        })
+        modules[0] = try compile(&session, root, &diagnostic_store)
+        errdefer ir.deinit(&modules[0])
+        modules[1] = try compile(&session, root, &diagnostic_store)
+    }
+    defer ir.deinit(&modules[0])
+    defer ir.deinit(&modules[1])
+    try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
+    for modules |module| {
+        outcome := try interpreter.run(mem.c_allocator, &module, &diagnostic_store)
+        defer interpreter.deinit(&outcome)
+        try testing.expect(outcome.trap == null)
+        try testing.expect_equal(i64(42), outcome.value.data.integer)
+    }
+}

source/sema/specialize/specialize.hon

@@ -681,6 +681,28 @@ expand_expected proc(state @mut State, id, expected ast.NodeId) void ! Error {
             try refresh(state)
             return
         }
+        if (view.operator == .equal or view.operator == .not_equal) {
+            operand := no_node
+            if (ast.expr_kind(&state.tree, view.left) == .literal_null) operand = view.right
+            else if (ast.expr_kind(&state.tree, view.right) == .literal_null) operand = view.left
+            if (operand != no_node) {
+                annotation :: try expression_type(state, operand, 0)
+                if (type_record_storage(state, annotation, 0)
+                    and ast.expr_kind(&state.tree, annotation) == .optional_type) {
+                    value :: try evaluate_expected(state, operand, annotation)
+                    if (value.kind != .aggregate) {
+                        try reject_evaluation(state, id)
+                        return
+                    }
+                    absent :: ast.expr_kind(&state.tree, value.node) == .literal_null
+                    folded :: try ast.add_boolean(state.builder, view.span,
+                        absent == (view.operator == .equal))
+                    try replace(state, id, folded)
+                    try refresh(state)
+                    return
+                }
+            }
+        }
         if (
             (view.operator == .equal or view.operator == .not_equal)
             and branch_constant(state, view.left, 0) and branch_constant(state, view.right, 0)
@@ -1789,7 +1811,7 @@ type_storage_annotation proc(tree @ast.Tree, bindings []sema.Binding, input ast.
             and type_storage_annotation(tree, bindings, view.initializer, depth + 1))
     }
     kind :: ast.expr_kind(tree, input)
-    if (kind == .alias_expr)
+    if (kind == .alias_expr or kind == .optional_type)
         return type_storage_annotation(tree, bindings, ast.type_operand(tree, input), depth + 1)
     if (kind == .array_type)
         return type_storage_annotation(tree, bindings, ast.array_type(tree, input).element, depth + 1)
@@ -3488,12 +3510,21 @@ type_match_arm proc(state @mut State, id ast.NodeId) usize ! Error {
     dispatch := subject
     if (metadata) {
         value :: try evaluate_expected(state, subject, annotation)
-        if (value.kind != .aggregate or ast.expr_kind(&state.tree, value.node) != .record_literal) return no_index
-        active :: ast.field(&state.tree, ast.record_field(&state.tree, value.node, 0))
+        if (value.kind != .aggregate) return no_index
         span :: ast.node_span(&state.tree, subject)
+        if (ast.expr_kind(&state.tree, annotation) == .optional_type) {
+            present :: ast.expr_kind(&state.tree, value.node) != .literal_null
+            boolean :: try ast.add_boolean(state.builder, span, true)
+            payload :: if (present) boolean else try ast.add_literal(state.builder, .null, span)
+            optional :: try ast.add_optional_type(state.builder, span, try builtin(state, subject, .bool))
+            dispatch = try ast.add_typed_constant(state.builder, span, optional, payload)
+        } else {
+            if (ast.expr_kind(&state.tree, value.node) != .record_literal) return no_index
+            active :: ast.field(&state.tree, ast.record_field(&state.tree, value.node, 0))
             tags :: try metadata_tags(state, annotation)
             tag :: try ast.add_enum_literal(state.builder, span, active.name?)
             dispatch = try ast.add_typed_constant(state.builder, span, tags, tag)
+        }
         try refresh(state)
     }
     for (0..ast.match_arm_count(&state.tree, id)) |i| {
@@ -3514,7 +3545,7 @@ type_match_arm proc(state @mut State, id ast.NodeId) usize ! Error {
             try expand(state, pattern)
             try arraylist.append(&patterns, pattern)
         }
-        if (metadata and arm.capture != null) {
+        if (metadata and !optional_capture and arm.capture != null) {
             expected := no_node
             valid := !arm.catch_all and patterns.items.len > 0
             for (patterns.items) |pattern| {
@@ -3570,7 +3601,11 @@ type_match_capture proc(state @mut State, id ast.NodeId) ast.NodeId ! Error {
             ast.node_span(&state.tree, subject), .unwrap, subject)
         try refresh(state)
         child :: try expression_type(state, unwrapped, 0)
-        return try type_execution_value(state, unwrapped, child)
+        payload :: try type_execution_value(state, unwrapped, child)
+        if (payload == no_node or !is_type(state, child)) return payload
+        result :: try ast.add_typed_constant(state.builder, ast.node_span(&state.tree, subject), child, payload)
+        try refresh(state)
+        return result
     }
     annotation :: try expression_type(state, subject, 0)
     value :: if (type_record_storage(state, annotation, 0))
@@ -3669,6 +3704,28 @@ expand_type_matches proc(state @mut State, id ast.NodeId) bool ! Error {
 type_array_projection proc(state @mut State, id ast.NodeId) bool ! Error {
     if (ast.node_class(&state.tree, id) != .expression) return true
     kind :: ast.expr_kind(&state.tree, id)
+    if (kind == .unary and ast.unary_expr(&state.tree, id).operator == .unwrap) {
+        operand :: ast.unary_expr(&state.tree, id).operand
+        annotation :: try expression_type(state, operand, 0)
+        if (!type_record_storage(state, annotation, 0)) return true
+        type :: try canonical_type(state, annotation)
+        if (type == no_node or ast.expr_kind(&state.tree, type) != .optional_type) return true
+        value :: try type_execution_value(state, operand, type)
+        if (value == no_node or !try type_array_initialized(state, value, id)) return false
+        payload :: ast.typed_constant(&state.tree, value).value
+        if (ast.expr_kind(&state.tree, payload) == .literal_null) {
+            state.evaluation_diagnostic = try diagnostics.add(state.diagnostics,
+                .null_unwrap, ast.node_span(&state.tree, id))
+            return false
+        }
+        selected :: ast.call_arg(&state.tree, payload, 0)
+        resolved :: binding(state, selected)
+        try remember(state, id)
+        ast.replace_node_from_source(state.builder, id, selected)
+        try refresh(state)
+        state.bindings.items[usize(id)] = resolved
+        return true
+    }
     if (kind == .index or kind == .selection) {
         base :: if (kind == .index) ast.index_expr(&state.tree, id).base else ast.selection_expr(&state.tree, id).base
         base_binding :: binding(state, base)
@@ -3845,6 +3902,8 @@ type_record_check proc(state @mut State, annotation ast.NodeId) bool ! Error {
     if (is_type(state, annotation)) return true
     type :: try canonical_type(state, annotation)
     if (type == no_node) return false
+    if (ast.expr_kind(&state.tree, type) == .optional_type)
+        return try type_record_check(state, ast.type_operand(&state.tree, type))
     if (ast.expr_kind(&state.tree, type) == .slice_type)
         return try type_record_check(state, ast.slice_type(&state.tree, type).element)
     if (ast.expr_kind(&state.tree, type) == .array_type) {
@@ -4010,6 +4069,8 @@ type_array_initialized proc(state @mut State, value, at ast.NodeId) bool ! Error
     }
     if (kind == .typed_constant)
         return try type_array_initialized(state, ast.typed_constant(&state.tree, value).value, at)
+    if (kind == .builtin_call and contextual_literal(state, value))
+        return try type_array_initialized(state, ast.call_arg(&state.tree, value, 0), at)
     if (kind == .array_literal) {
         for (0..ast.array_element_count(&state.tree, value)) |i| {
             if (!try type_array_initialized(state, ast.array_element(&state.tree, value, i), at)) return false
@@ -4248,6 +4309,42 @@ type_array_update proc(
     return result
 }
 
+# optional metadata uses the same explicit presence encoding as serialized values.
+# an inner null is a payload only when wrapped in some!, never outer absence.
+@hide:file
+type_optional_value proc(state @mut State, expression, type ast.NodeId) ast.NodeId ! Error {
+    if (!try type_record_check(state, type)) return no_node
+    kind :: ast.expr_kind(&state.tree, expression)
+    if (kind == .literal_undefined or kind == .literal_null) {
+        result :: try ast.add_typed_constant(state.builder, ast.node_span(&state.tree, expression), type, expression)
+        try refresh(state)
+        return result
+    }
+    child :: ast.type_operand(&state.tree, type)
+    input := expression
+    if (kind == .typed_constant) {
+        actual :: ast.typed_constant(&state.tree, expression)
+        if (try same_type(state, type, actual.type)) return expression
+    } else if (kind == .builtin_call
+        and mem.eql(ast.literal_text(&state.tree, ast.builtin_call(&state.tree, expression).name), "some")) {
+        if (ast.call_arg_count(&state.tree, expression) != 1) {
+            state.evaluation_diagnostic = try diagnostics.add(state.diagnostics,
+                .argument_count_mismatch, ast.node_span(&state.tree, expression))
+            return no_node
+        }
+        input = ast.call_arg(&state.tree, expression, 0)
+    }
+    payload :: try type_execution_value(state, input, child)
+    if (payload == no_node) return no_node
+    span :: ast.node_span(&state.tree, expression)
+    name :: try metadata_member(state, expression, "some")
+    arguments [1]ast.NodeId :: [payload]
+    present :: try ast.add_builtin_call(state.builder, span, name, arguments[..])
+    result :: try ast.add_typed_constant(state.builder, span, type, present)
+    try refresh(state)
+    return result
+}
+
 @hide:file
 type_execution_value proc(state @mut State, input, annotation ast.NodeId) ast.NodeId ! Error {
     if (input == no_node or annotation == no_node) return no_node
@@ -4311,6 +4408,11 @@ type_execution_value proc(state @mut State, input, annotation ast.NodeId) ast.No
             .type_mismatch, ast.node_span(&state.tree, expression))
         return no_node
     }
+    if (!is_type(state, annotation) and type_record_storage(state, annotation, 0)) {
+        type :: try canonical_type(state, annotation)
+        if (type != no_node and ast.expr_kind(&state.tree, type) == .optional_type)
+            return try type_optional_value(state, expression, type)
+    }
     if (!is_type(state, annotation) and type_record_storage(state, annotation, 0))
         return try type_record_value(state, expression, annotation)
     if (is_type(state, annotation)) {
@@ -4370,6 +4472,16 @@ type_capture_guard proc(state @mut State, expression, capture ast.NodeId) Value
         state.evaluation_diagnostic = resolved.data.diagnostic
         return Value{}
     }
+    annotation :: try expression_type(state, expression, 0)
+    if (type_record_storage(state, annotation, 0)) {
+        type :: try canonical_type(state, annotation)
+        if (type != no_node and ast.expr_kind(&state.tree, type) == .optional_type) {
+            value :: try type_execution_value(state, expression, type)
+            if (value == no_node or !try type_array_initialized(state, value, expression)) return Value{}
+            return Value{ kind = .boolean, node = expression,
+                boolean = ast.expr_kind(&state.tree, ast.typed_constant(&state.tree, value).value) != .literal_null }
+        }
+    }
     span :: ast.node_span(&state.tree, expression)
     yes :: try ast.add_boolean(state.builder, span, true)
     no :: try ast.add_boolean(state.builder, span, false)

tests/native.py

@@ -2278,6 +2278,143 @@ CASES = {
         }
         """, b"", b"initialization",
     ),
+    "optional_types_procedure_boundaries_copies_and_static_captures": (
+        """
+        Token :: distinct i16
+        choose proc(present bool) ?type {
+            if (present) return Token
+            return null
+        }
+        pass proc(value ?type) ?type { return value }
+        Box proc(value ?type) type {
+            T :: if value |Child| Child else bool
+            return struct { item T }
+        }
+        Selected proc(value ?type) type { return value? }
+        exact proc($value ?type, input Selected(value)) Selected(value) { return input }
+        Storage proc() type {
+            selected ?type := pass(choose(true))
+            saved :: selected
+            selected = null
+            if (selected != null or null == saved) return bool
+            if (Box(saved) != Box(choose(true))) return bool
+            if (Box(null) == Box(saved)) return bool
+            while saved |T| {
+                return [2]T
+            }
+            return bool
+        }
+        main proc() i16 {
+            values Storage() :: [40, 2]
+            return i16(exact(choose(true), values[0])) + i16(values[1])
+        }
+        """, b"42\n", None,
+    ),
+    "optional_metadata_records_nested_presence_and_owned_updates": (
+        """
+        Config :: struct { child type, count usize }
+        config proc() ?Config { return { child = i16, count = 2 } }
+        nested proc(present bool) ??type {
+            if (present) return some!(null)
+            return null
+        }
+        Storage proc() type {
+            outer :: nested(true)
+            if outer |inner| {
+                if (inner != null or nested(false) != null) return bool
+            } else return bool
+            values [2]mut ?Config := [config(), null]
+            saved :: values
+            values[0] = null
+            values[1] = { child = bool, count = 3 }
+            if (values[0] != null or saved[1] != null) return bool
+            info :: saved[0]?
+            return [info.count]info.child
+        }
+        main proc() Storage() { return [40, 2] }
+        """, b"[40, 2]\n", None,
+    ),
+    "optional_types_fill_undefined_fields_and_yield_before_cleanup": (
+        """
+        Config :: struct { child ?type = bool, count usize }
+        make proc() ?Config {
+            config Config := undefined
+            config.child = i16
+            config.count = 2
+            return some!(config)
+        }
+        Storage proc() type {
+            value ?type := undefined
+            value = null
+            selected ?type :: {
+                defer value = bool
+                yield make()?.child
+            }
+            if (value? != bool) return bool
+            T :: selected?
+            return [make()?.count]T
+        }
+        main proc() Storage() { return [40, 2] }
+        """, b"[40, 2]\n", None,
+    ),
+    "optional_types_reject_null_unwrap": (
+        """
+        empty proc() ?type { return null }
+        Bad proc() type { return empty()? }
+        main proc() Bad() { return 42 }
+        """, b"", b"cannot unwrap a null optional",
+    ),
+    "optional_types_reject_non_type_payload": (
+        """
+        Bad proc() ?type { return some!(42) }
+        Selected proc() type { return Bad()? }
+        main proc() Selected() { return 42 }
+        """, b"", b"incompatible value type",
+    ),
+    "optional_types_reject_incomplete_payload_result": (
+        """
+        Config :: struct { child type, count usize }
+        bad proc() ?Config {
+            value Config := undefined
+            value.child = i16
+            return some!(value)
+        }
+        Selected proc() type { return bad()?.child }
+        main proc() Selected() { return 42 }
+        """, b"", b"read before initialization",
+    ),
+    "optional_types_reject_uninitialized_presence": (
+        """
+        Bad proc() type {
+            value ?type := undefined
+            if value |_| return i16
+            return bool
+        }
+        main proc() Bad() { return 42 }
+        """, b"", b"read before initialization",
+    ),
+    "optional_types_reject_capture_mutation": (
+        """
+        Bad proc() type {
+            value ?type :: i16
+            if value |T| { T = bool }
+            return i16
+        }
+        main proc() Bad() { return 42 }
+        """, b"", b"mutable local binding",
+    ),
+    "optional_types_reject_runtime_dependency": (
+        """
+        choose proc(present bool) ?type {
+            if (present) return i16
+            return null
+        }
+        main proc() i16 {
+            present := true
+            return if choose(present) |_| 42 else 0
+        }
+        """, b"", b"compile-time value",
+    ),
     "type_records_undefined_nested_updates_preserve_partial_copies": (
         """
         Config :: struct {