feat: materialize nonzero-sentinel compile-time byte slices
Own nonzero trailing sentinels in canonical string-backed snapshots and preserve specialization identity across computed and direct views. Verified: bro build; bro test (296 passed); python3 tests/native.py (1026 interpreter/native comparisons at -O0/-O2, 5 native allocator programs, LLVM output and CLI errors).
This commit is contained in:
@@ -136,7 +136,7 @@ ordinary procedures may compute integer, boolean, float and string compile-time
|
||||
|
||||
computed callbacks materialize as references to their original source declarations, never as temporary interpreter function IDs. `choose proc() @proc(value i16) i16 { return increment }` can supply `$(choose())`, compile-time arguments and generated defaults. optional callbacks preserve absence and nested presence; arrays, records, tagged payloads and both fallible channels retain exact callback signatures and specialization-cache identity. imported private targets remain bound to their originating declarations, including across repeated compilation and native emission.
|
||||
|
||||
readonly byte-slice results (`[]u8` and zero-terminated `[;0]u8`) snapshot their bytes into owned literals before evaluation storage is released. ordinary compile-time helpers can return string subviews, views of locally filled arrays or immutable global arrays, including nested records/arrays, tagged payloads, optionals and both fallible channels. explicit `$()` expressions, generic arguments and generated defaults preserve embedded NULs, arbitrary byte values, exact declared slice types and content-based specialization-cache identity. snapshots survive evaluation/session disposal and repeated compilation; uninitialized source bytes diagnose read-before-initialization. mutable slices, nonbyte or nominal-byte slices and nonzero-sentinel slices remain unsupported.
|
||||
readonly byte-slice results (`[]u8` and `[;sentinel]u8` for any byte sentinel) snapshot their bytes into owned literals before evaluation storage is released. ordinary compile-time helpers can return string subviews, views of locally filled arrays or immutable global arrays, including nested records/arrays, tagged payloads, optionals and both fallible channels. explicit `$()` expressions, generic arguments and generated defaults preserve embedded NULs, arbitrary byte values, exact declared slice types and content-based specialization-cache identity. nonzero-sentinel snapshots retain an owned trailing sentinel outside the logical length; empty views also retain it. snapshots survive evaluation/session disposal and repeated compilation; uninitialized source bytes diagnose read-before-initialization, and sentinel bytes are rechecked after evaluation. mutable slices and nonbyte or nominal-byte slices remain unsupported.
|
||||
|
||||
`$(expression)` explicitly evaluates a constant expression, including an ordinary procedure call, even in a package without templates. use a typed expression when its shape cannot be inferred independently, such as `$(Point { x = 40 })`; integer literals retain contextual typing. the resulting value works in runtime expressions, bindings, expression statements and array lengths, while the erased expression still receives ordinary checking. each interpreter evaluation is isolated and limited to 100,000 IR instructions/terminators and 128 active frames; failures retain their originating source locations. runtime parameters and mutable local/global dependencies cannot supply compile-time inputs. nonnull data pointers, unsupported slice types and untagged unions cannot escape temporary evaluation as generated constants; active payloads or aggregate fields containing unsupported values are diagnosed rather than retaining interpreter handles. untagged unions lack an active-member identity, so selecting a source variant could discard observable bytes. executed foreign calls remain interpreter traps. general type-valued execution, broader storage-bearing materialization and remaining existing-library acceptance, including broader `meta` operations and `io.print`, remain open.
|
||||
|
||||
|
||||
@@ -22,9 +22,9 @@ compile-time arguments and generated record/array storage through both paths.
|
||||
ordinary procedures can compute scalar, nested record/array, tagged-union,
|
||||
fallible, callback and readonly byte-slice compile-time values using mutable locals, loops and recursive calls;
|
||||
explicit `$()` expressions use the same constant evaluation path.
|
||||
readonly `[]u8` and zero-terminated `[;0]u8` snapshots own exact bytes after evaluation.
|
||||
mutable, nonbyte/nominal-byte and nonzero-sentinel slices, nonnull data pointers
|
||||
and untagged unions retain the constant-materialization escape boundary.
|
||||
readonly `[]u8` and `[;sentinel]u8` snapshots own exact bytes and any byte sentinel after evaluation.
|
||||
mutable and nonbyte/nominal-byte slices, nonnull data pointers and untagged unions
|
||||
retain the constant-materialization escape boundary.
|
||||
compile-time `if`/`match` selection and type comparisons specialize only the selected body,
|
||||
including owned by-value tagged payload captures in ordinary generic procedures.
|
||||
type-producing procedures also accept ordinary
|
||||
@@ -146,8 +146,9 @@ test discovery and FFI remain outside these completed slices.
|
||||
- [x] hash-map execution checkpoint: existing `std/hashmap.HashMap`, `AutoHashMap` and `StringHashMap` run collision chains, zero-hash normalization, growth/rehashing, duplicate-key errors, absent lookups and owned aggregate values without replacement library code. interpreter checks use separate typed allocation buffers and preserve the original map on allocation failure, including cleanup; native `mem.c_allocator` checks cover all integer widths/boundaries, booleans, sparse/aliased enums, nested distinct/array/record keys with padding ignored, content-based byte strings and custom callbacks at `-O0`/`-O2`. specialization retains `usize` collection indices, nominal/explicit pointer-cast result types, rigid slice borrowing/string context and source-known optional callbacks; ordinary checking rejects incompatible elements and readonly reversal. `unreachable` demand-traps with provenance through both paths and compile-time execution; literal-true loops distinguish escaping and nested breaks. `compile_error!` owns computed string diagnostics and skips unselected compile-time bodies. initializer inference from expected result types is covered below.
|
||||
- [x] expected-result inference checkpoint: omitted compile-time type/value arguments infer from rigid runtime arguments and expected results before contextual literal defaults. typed bindings, assignments, returns, ordinary/nested calls, aggregate fields/elements/defaults, optional/fallible channels, callbacks, match arms, yielding/catch blocks, arithmetic/bitwise operands and explicit `$()` expressions retain exact checked types through interpreter/native execution. generated nominal results infer counts and private/custom callback identities; existing `hashmap.init(allocator)` initializes `AutoHashMap`, `StringHashMap` and custom `HashMap` without replacement library code. rigid conflicts, unrelated nominal generators and unresolved arguments diagnose; explicit calls and fully explicit type-producer inputs retain their contracts. owned specializations and repeat compilation are verified.
|
||||
- [x] computed-callback materialization checkpoint: ordinary compile-time calls return source-bound callbacks through interpreter/native execution, including optional/nested presence, arrays/records, tagged payloads and both fallible channels. temporary IR function IDs map back to exact source declarations before evaluation storage is released. computed/direct aggregate arguments share specialization-cache identity and generated defaults; imported private targets, same-spelled functions, signature diagnostics, runtime trap provenance and repeated compilation retain their contracts. nonnull data pointers, unsupported slice types and untagged unions retain their escape boundary.
|
||||
- [x] readonly byte-slice materialization checkpoint: ordinary compile-time calls snapshot readonly `[]u8` and zero-terminated `[;0]u8` results into owned literals before evaluation storage is released. string subviews, locally filled arrays, immutable global arrays and raw aggregate views preserve exact bytes, embedded NULs, declared slice types and normalized content-based cache identity through explicit `$()` expressions, generic inputs, generated defaults, nested records/arrays, tagged payloads, optionals and both fallible channels. imported helpers, session/source disposal and repeated compilation retain ownership through interpreter/native execution; uninitialized source bytes diagnose read-before-initialization and invalid ranges are rejected before dereference. mutable, nonbyte/nominal-byte and nonzero-sentinel slices, nonnull data pointers and untagged unions remain unsupported.
|
||||
- [ ] remaining gate: general type-valued compile-time execution, broader constant materialization (including untagged unions, nonnull data pointers and slice types beyond readonly plain/zero-terminated bytes), plus remaining existing-library acceptance for broader `meta` operations and `io.print`. formatting parsing and broader imported C-binding support may interleave with later gates.
|
||||
- [x] readonly byte-slice materialization checkpoint: ordinary compile-time calls snapshot readonly `[]u8` and zero-terminated `[;0]u8` results into owned literals before evaluation storage is released. string subviews, locally filled arrays, immutable global arrays and raw aggregate views preserve exact bytes, embedded NULs, declared slice types and normalized content-based cache identity through explicit `$()` expressions, generic inputs, generated defaults, nested records/arrays, tagged payloads, optionals and both fallible channels. imported helpers, session/source disposal and repeated compilation retain ownership through interpreter/native execution; uninitialized source bytes diagnose read-before-initialization and invalid ranges are rejected before dereference. nonzero sentinels are covered below; mutable and nonbyte/nominal-byte slices, nonnull data pointers and untagged unions remain unsupported.
|
||||
- [x] nonzero-sentinel byte-slice checkpoint: readonly `[;sentinel]u8` compile-time results own exact payload bytes and any byte sentinel through interpreter/native execution. canonical string-backed views preserve logical lengths, empty views, embedded NULs, high bytes, computed/direct specialization-cache identity, generated defaults, inline captures, nested aggregates, optionals and both fallible channels. sentinel checks observe final evaluation storage, diagnosing mutation, uninitialized bytes and invalid addresses; imported helpers, source/session disposal and repeated compilation retain ownership. mutable and nonbyte/nominal-byte slices, nonnull data pointers and untagged unions retain the escape boundary.
|
||||
- [ ] remaining gate: general type-valued compile-time execution, broader constant materialization (including untagged unions, nonnull data pointers and slice types beyond readonly bytes), plus remaining existing-library acceptance for broader `meta` operations and `io.print`. formatting parsing and broader imported C-binding support may interleave with later gates.
|
||||
- [ ] remaining source constructs: multiline strings. boolean and optional-capture conditional expressions, yielding value blocks, `yield`, compound updates and bitwise operators are covered above. examples: `source/parser/parse.hon`, `source/llvm/runtime.hon`.
|
||||
- [ ] standard-library milestone: compile/run a multi-package program exercising existing allocation and IO, including error cleanup; no replacement miniature standard library.
|
||||
- [ ] native driver integration: required C ABI declarations, `process.Init` startup arguments/IO, actual CLI exit statuses, file loading, temporary files and spawning Clang. Honey-generated programs currently accept only parameterless `main` and print its result; Bro's bootstrap startup support is not Honey backend support.
|
||||
|
||||
@@ -965,6 +965,8 @@ computed_byte_slices_recompile_without_borrowing_evaluation_storage test {
|
||||
` if (packet.bytes[2] != 128 or packet.bytes[3] != 42) return -3
|
||||
` if (packet.suffix[0] != 65 or packet.suffix[1] != 0) return -4
|
||||
` if (packet.suffix[2] != 66 or packet.suffix.ptr[3] != 0) return -5
|
||||
` if (packet.marked.len != 2 or packet.marked.ptr[2] != 255) return -6
|
||||
` if (packet.marked[0] != 40 or packet.marked[1] != 2) return -7
|
||||
` return 42
|
||||
`}
|
||||
,
|
||||
@@ -972,14 +974,15 @@ computed_byte_slices_recompile_without_borrowing_evaluation_storage test {
|
||||
defer arraylist.deinit(&bytes)
|
||||
root_source :: try add_source(&session, root, "/byte_slices/app/main.hon", &bytes, &diagnostic_store)
|
||||
imported := try owned_text(
|
||||
`Packet :: struct { bytes []u8, suffix [;0]u8 }
|
||||
`Packet :: struct { bytes []u8, suffix [;0]u8, marked [;255]u8 }
|
||||
`make proc() Packet {
|
||||
` bytes [4]mut u8 := undefined
|
||||
` bytes[0] = 255
|
||||
` bytes[1] = 0
|
||||
` bytes[2] = 128
|
||||
` bytes[3] = 42
|
||||
` return Packet { bytes = bytes[..], suffix = "xA\0B"[1..;0] }
|
||||
` marked [3]mut u8 := [40, 2, 255]
|
||||
` return Packet { bytes = bytes[..], suffix = "xA\0B"[1..;0], marked = marked[..2;255] }
|
||||
`}
|
||||
,
|
||||
)
|
||||
|
||||
@@ -2325,6 +2325,7 @@ evaluate_expected proc(state @mut State, id, annotation ast.NodeId) Value ! Erro
|
||||
and !try type_array_initialized(state, id, id)) return Value{}
|
||||
if (type != no_node and (contextual_literal(state, id)
|
||||
or ast.expr_kind(&state.tree, id) == .block_expr
|
||||
or ast.expr_kind(&state.tree, id) == .slice
|
||||
or ast.expr_kind(&state.tree, type) == .fallible_type))
|
||||
return try evaluate_ir(state, id, type)
|
||||
}
|
||||
@@ -2332,7 +2333,7 @@ evaluate_expected proc(state @mut State, id, annotation ast.NodeId) Value ! Erro
|
||||
}
|
||||
|
||||
# values leaving temporary evaluation must own no interpreter addresses or module IDs.
|
||||
# snapshot readonly bytes and rebuild source literals/references before releasing its backing.
|
||||
# snapshot readonly bytes and rebuild source literals/references before releasing their backing.
|
||||
@hide:file
|
||||
serialize_value proc(
|
||||
state @mut State,
|
||||
@@ -2358,10 +2359,7 @@ serialize_value proc(
|
||||
result := no_node
|
||||
if (type.kind == .slice) {
|
||||
slice :: typecheck.array_type(checked, type.array_type)
|
||||
if (
|
||||
slice.mutable or slice.element.kind != .u8 or slice.element.nominal != no_node
|
||||
or (slice.sentinel != null and slice.sentinel? != 0)
|
||||
) return no_node
|
||||
if (slice.mutable or slice.element.kind != .u8 or slice.element.nominal != no_node) return no_node
|
||||
pointer :: ir.aggregate_item(module, value, 0)
|
||||
length :: ir.aggregate_item(module, value, 1)
|
||||
if (pointer.uninitialized or length.uninitialized) {
|
||||
@@ -2372,9 +2370,13 @@ serialize_value proc(
|
||||
length.data.unsigned != 0
|
||||
and interpreter.evaluation_byte(outcome, module, pointer, usize(length.data.unsigned - 1)) == null
|
||||
) return no_node
|
||||
bytes :: try mem.alloc(u8, state.allocator, usize(length.data.unsigned))
|
||||
nonzero_sentinel :: slice.sentinel != null and slice.sentinel? != 0
|
||||
length_bytes :: usize(length.data.unsigned)
|
||||
if (nonzero_sentinel and length_bytes == maxval!(usize)) return no_node
|
||||
extra :: if (nonzero_sentinel) usize(1) else usize(0)
|
||||
bytes :: try mem.alloc(u8, state.allocator, length_bytes + extra)
|
||||
defer mem.free(state.allocator, bytes)
|
||||
for (bytes) |_, index| {
|
||||
for (0..length_bytes) |index| {
|
||||
byte :: interpreter.evaluation_byte(outcome, module, pointer, index)
|
||||
if (byte == null) return no_node
|
||||
if (byte?.uninitialized) {
|
||||
@@ -2384,18 +2386,24 @@ serialize_value proc(
|
||||
bytes[index] = u8(byte?.data.unsigned)
|
||||
}
|
||||
if (slice.sentinel != null) {
|
||||
sentinel :: interpreter.evaluation_byte(outcome, module, pointer, bytes.len)
|
||||
sentinel :: interpreter.evaluation_byte(outcome, module, pointer, length_bytes)
|
||||
if (sentinel == null) return no_node
|
||||
if (sentinel?.uninitialized) {
|
||||
state.evaluation_diagnostic = try diagnostics.add(state.diagnostics, .read_before_initialization, span)
|
||||
return no_node
|
||||
}
|
||||
if (sentinel?.data.unsigned != 0) {
|
||||
if (sentinel?.data.unsigned != u64(slice.sentinel?)) {
|
||||
state.evaluation_diagnostic = try diagnostics.add(state.diagnostics, .sentinel_mismatch, span)
|
||||
return no_node
|
||||
}
|
||||
if (nonzero_sentinel) bytes[length_bytes] = u8(slice.sentinel?)
|
||||
}
|
||||
result = try ast.add_string(state.builder, span, bytes)
|
||||
if (nonzero_sentinel) {
|
||||
end :: try ast.add_integer(state.builder, span, u64(length_bytes))
|
||||
sentinel :: try ast.add_integer(state.builder, span, u64(slice.sentinel?))
|
||||
result = try ast.add_slice(state.builder, span, result, null, end, sentinel)
|
||||
}
|
||||
} else if (type.kind == .array) {
|
||||
array :: typecheck.array_type(checked, type.array_type)
|
||||
elements := arraylist.init(ast.NodeId, state.allocator)
|
||||
@@ -2510,6 +2518,16 @@ constant_equal proc(state @mut State, left, right ast.NodeId) bool ! Error {
|
||||
b :: ast.typed_constant(&state.tree, right)
|
||||
return try same_type_context(state, a.type, b.type, true) and try constant_equal(state, a.value, b.value)
|
||||
}
|
||||
if (kind == .slice) {
|
||||
a :: ast.slice_expr(&state.tree, left)
|
||||
b :: ast.slice_expr(&state.tree, right)
|
||||
# serialized nonzero-sentinel views have a canonical owned string and explicit end.
|
||||
return (
|
||||
try constant_equal(state, a.base, b.base)
|
||||
and try constant_equal(state, a.end?, b.end?)
|
||||
and try constant_equal(state, a.sentinel?, b.sentinel?)
|
||||
)
|
||||
}
|
||||
if (kind == .array_literal) {
|
||||
count :: ast.array_element_count(&state.tree, left)
|
||||
if (count != ast.array_element_count(&state.tree, right)) return false
|
||||
@@ -2547,6 +2565,11 @@ constant_hash proc(state @mut State, id ast.NodeId) usize ! Error {
|
||||
value :: ast.typed_constant(&state.tree, id)
|
||||
return mix(mix(hash, try type_hash_context(state, value.type, true)), try constant_hash(state, value.value))
|
||||
}
|
||||
if (kind == .slice) {
|
||||
view :: ast.slice_expr(&state.tree, id)
|
||||
return mix(mix(mix(hash, try constant_hash(state, view.base)),
|
||||
try constant_hash(state, view.end?)), try constant_hash(state, view.sentinel?))
|
||||
}
|
||||
if (kind == .array_literal) {
|
||||
result := hash
|
||||
for (0..ast.array_element_count(&state.tree, id)) |i|
|
||||
@@ -2684,6 +2707,7 @@ evaluate_ir_body proc(state @mut State, at, body, result_type ast.NodeId) Value
|
||||
}
|
||||
if (
|
||||
(ir.aggregate(value.type) and source_type.kind != .slice)
|
||||
or (source_type.kind == .slice and ast.expr_kind(&state.tree, node) == .slice)
|
||||
or source_type.kind == .enumeration or source_type.kind == .void
|
||||
)
|
||||
return Value{ kind = .aggregate, node = node, annotation = result_type }
|
||||
|
||||
+120
-4
@@ -5418,11 +5418,127 @@ CASES = {
|
||||
main proc() usize { return $(make()).len }
|
||||
""", b"", b"compile-time result cannot escape evaluation",
|
||||
),
|
||||
"comptime_aggregate_rejects_nonzero_sentinel_byte_slice_escape": (
|
||||
"comptime_nonzero_sentinel_byte_slices_own_subviews": (
|
||||
"""
|
||||
make proc() [;1]u8 {
|
||||
values [2]u8 :: [42, 1]
|
||||
return values[..1;1]
|
||||
make proc() [;255]u8 {
|
||||
values [5]mut u8 := [9, 40, 0, 2, 255]
|
||||
return values[1..4;255]
|
||||
}
|
||||
empty proc() [;128]u8 {
|
||||
values [1]u8 :: [128]
|
||||
return values[..0;128]
|
||||
}
|
||||
main proc() usize {
|
||||
value :: $(make())
|
||||
zero :: $(empty())
|
||||
if (value.len != 3 or value[1] != 0 or value.ptr[3] != 255) return 1
|
||||
if (zero.len != 0 or zero.ptr[0] != 128) return 2
|
||||
return usize(value[0]) + usize(value[2])
|
||||
}
|
||||
""", b"42\n", None,
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_normalize_cache_and_defaults": (
|
||||
"""
|
||||
make proc(padded bool) [;7]u8 {
|
||||
if (padded) {
|
||||
prefixed [4]u8 :: [9, 40, 2, 7]
|
||||
return prefixed[1..3;7]
|
||||
}
|
||||
values [3]u8 :: [40, 2, 7]
|
||||
return values[..2;7]
|
||||
}
|
||||
Box proc($bytes [;7]u8) type { return struct { bytes [;7]u8 = bytes } }
|
||||
main proc() u8 {
|
||||
a Box(make(false)) :: {}
|
||||
b Box(make(true)) :: a
|
||||
direct Box("!(\\x02\\x07"[1..3;7]) :: b
|
||||
if (direct.bytes.len != 2 or direct.bytes.ptr[2] != 7) return 0
|
||||
return direct.bytes[0] + direct.bytes[1]
|
||||
}
|
||||
""", b"42\n", None,
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_cross_nested_channels": (
|
||||
"""
|
||||
View :: struct { rows [1]??[;9]u8 }
|
||||
Choice :: union(enum) { view View, empty void }
|
||||
Failure :: struct { bytes [;9]u8 }
|
||||
make proc() [;9]u8 {
|
||||
values [3]u8 :: [40, 2, 9]
|
||||
return values[..2;9]
|
||||
}
|
||||
choice proc() Choice { return .view { rows = [some!(some!(make()))] } }
|
||||
success proc() View ! Failure { return { rows = [some!(some!(make()))] } }
|
||||
failure proc() View ! Failure { return Failure { bytes = make() } }
|
||||
main proc() usize {
|
||||
value :: $(choice())
|
||||
good :: $(success()) catch { return 1 }
|
||||
bad :: $(failure()) catch |err| {
|
||||
match value {
|
||||
.view |view|: {
|
||||
bytes :: view.rows[0]??
|
||||
if (bytes.ptr[2] != 9 or good.rows[0]??.ptr[2] != 9) return 2
|
||||
return usize(bytes[0]) + usize(err.bytes[1])
|
||||
}
|
||||
.empty: return 3
|
||||
}
|
||||
}
|
||||
return bad.rows.len
|
||||
}
|
||||
""", b"42\n", None,
|
||||
),
|
||||
"inline_for_materializes_nonzero_sentinel_byte_slices": (
|
||||
"""
|
||||
make proc() [1][;1]u8 {
|
||||
bytes [2]u8 :: [42, 1]
|
||||
return [bytes[..1;1]]
|
||||
}
|
||||
main proc() u8 {
|
||||
inline for make() |bytes| {
|
||||
if (bytes.ptr[1] != 1) return 0
|
||||
return bytes[0]
|
||||
}
|
||||
return 0
|
||||
}
|
||||
""", b"42\n", None,
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_recheck_mutated_sentinel": (
|
||||
"""
|
||||
make proc() [;255]u8 {
|
||||
values [2]mut u8 := [42, 255]
|
||||
view [;255]u8 :: values[..1;255]
|
||||
values[1] = 0
|
||||
return view
|
||||
}
|
||||
main proc() usize { return $(make()).len }
|
||||
""", b"", b"sentinel",
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_reject_uninitialized_sentinel": (
|
||||
"""
|
||||
make proc() [;7]u8 {
|
||||
values [2]mut u8 := [42, 7]
|
||||
view [;7]u8 :: values[..1;7]
|
||||
other [2]mut u8 := undefined
|
||||
other[0] = 42
|
||||
values = other
|
||||
return view
|
||||
}
|
||||
main proc() usize { return $(make()).len }
|
||||
""", b"", b"before initialization",
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_reject_out_of_bounds_sentinel": (
|
||||
"""
|
||||
make proc() [;7]u8 {
|
||||
values [1]u8 :: [42]
|
||||
return values.ptr[..1;7]
|
||||
}
|
||||
main proc() usize { return $(make()).len }
|
||||
""", b"", b"bounds",
|
||||
),
|
||||
"comptime_nonzero_sentinel_byte_slices_reject_mutable_escape": (
|
||||
"""
|
||||
make proc() [;7]mut u8 {
|
||||
values [2]mut u8 := [42, 7]
|
||||
return values[..1;7]
|
||||
}
|
||||
main proc() usize { return $(make()).len }
|
||||
""", b"", b"compile-time result cannot escape evaluation",
|
||||
|
||||
Reference in New Issue
Block a user