Compare commits

...

13 Commits

33 changed files with 1483 additions and 298 deletions
+1
View File
@@ -0,0 +1 @@
style: before writing or reviewing `.hon` code, read [docs/style.md](docs/style.md).
+12 -2
View File
@@ -1,9 +1,19 @@
# Honey
Honey is currently bootstrapped with the Bro compiler. The expected development workspace places the Bro project at [`../brolang`](../brolang); consult that project for compiler behavior and documentation.
Honey is currently bootstrapped with the Bro compiler. the expected development workspace places the Bro project at [`../brolang`](../brolang); consult that project for compiler behavior and documentation.
Build Honey from this directory:
style: see the [language guidelines](docs/style.md).
build Honey from this directory:
```sh
bro build
```
the parser supports arithmetic expressions with nested parenthesized groups, such as `a + (b - c) * 2 / 3`. groups control precedence without adding AST nodes; inner expression spans exclude the grouping parentheses.
function calls are postfix expressions: `f()`, `f(a, b)`, and `factory()(x)`. calls bind tighter than unary and binary arithmetic and can appear in initializers or as expression statements inside function bodies. argument lists allow a trailing comma; newlines remain statement boundaries. argument/type checking belongs to semantic analysis and is not yet implemented.
name resolution (`@source/sema`) runs after parsing. it predeclares package names and function-body locals, allowing forward references and mutually recursive functions. parameters and locals share a function scope; signatures resolve names in the enclosing package scope. shadowing is forbidden, including reuse of builtin type names (`int`, `float`, `string`, `void`); sibling function scopes may independently reuse names.
`sema.resolve` returns an owned binding table indexed by AST `NodeId`, without modifying the AST. `sema.binding` identifies a declaration, builtin type, or invalid binding carrying its diagnostic ID; nodes with no binding return `.none`. release the result with `sema.deinit`. unknown names and duplicate declarations add source-spanned diagnostics; invalid syntax retains its existing diagnostic identity. same-scope duplicates retain the first declaration for recovery, while forbidden inner declarations invalidate that name's uses rather than silently binding them to an outer declaration. type checking and read-before-initialization checks remain separate, unimplemented stages.
+4 -4
View File
@@ -1,6 +1,6 @@
b :: import "@std/build"
build :: import "@std/build"
config :: b.BuildConfig{
name = "honey",
source = "source",
config :: build.BuildConfig{
name = "honey",
source = "source",
}
+2 -2
View File
@@ -1,3 +1,3 @@
# Compile through recoverable source errors
# compile through recoverable source errors
Honey produces an executable whenever source errors can be represented safely: structurally valid syntax trees may contain typed invalid nodes carrying source spans and diagnostic identities, and semantic checking or lowering turns demanded invalid code into diagnostic runtime traps. Clean compilation returns status `0`, compilation that emits an executable with source diagnostics returns `1`, and unrecoverable infrastructure or backend failure returns `2`. This deliberately favors testing unaffected program paths over conventional fail-fast compilation; parser recovery must preserve valid surrounding syntax, while compact syntax-tree storage remains private behind typed views.
Honey produces an executable whenever source errors can be represented safely: structurally valid syntax trees may contain typed invalid nodes carrying source spans and diagnostic identities, and semantic checking or lowering turns demanded invalid code into diagnostic runtime traps. clean compilation returns status `0`, compilation that emits an executable with source diagnostics returns `1`, and unrecoverable infrastructure or backend failure returns `2`. this deliberately favors testing unaffected program paths over conventional fail-fast compilation; parser recovery must preserve valid surrounding syntax, while compact syntax-tree storage remains private behind typed views.
+70
View File
@@ -0,0 +1,70 @@
# Honey style
these guidelines apply to Honey programs, the compiler, and the standard library. the compiler is currently written in Bro; apply the same conventions using syntax supported by the bootstrap compiler. generated C bindings retain the generator's declarations and formatting.
## names
- use `snake_case` for procedures, variables, constants, fields, and enum members; `PascalCase` for types.
- choose names that describe the value or operation. short loop indices are appropriate when their meaning is local. preserve external ABI names.
- qualify imported operations with their module, such as `mem.alloc` and `ast.node_count`. mark file-local implementation details with `@hide:file`.
## bindings and types
- prefer `::` for bindings that stay fixed; use `:=` for bindings that change, including values mutated through pointers.
- infer types when the initializer makes the intended type clear. write an explicit type when it determines representation, mutability, or a public contract.
- request mutable pointers and slices only when the operation needs to modify their contents.
- give aggregates meaningful defaults. use `undefined` only when every value that can be read is assigned first.
## layout and control flow
- use four-space indentation and put opening braces on the declaration or control-flow line.
- use single-line bodies for simple operations. use braces for multiple statements or nested control flow.
- break long parameter lists and aggregate initializers across lines, with a trailing comma on the last item.
- when breaking boolean or arithmetic expressions across lines, put continuation operators at the start of each line. indent operands one level inside the enclosing parentheses and put the closing parenthesis on its own line.
- wrap multiline `if` and `while` guards in parentheses, with the opening body brace after the closing parenthesis, so the guard's end is visually distinct from its body.
```hon
debug.assert(
node.kind == .invalid_expr
or node.kind == .invalid_stmt
or node.kind == .invalid_decl
)
while (
!next_is(state, .close_paren)
and !next_is(state, .newline)
and !next_is(state, .close_curly)
and !next_is(state, .eof)
) {
argument :: try parse_expr(state)
try arraylist.append(&arguments, argument)
if (!next_is(state, .comma)) break
state.next_token += 1
}
```
- use early returns for terminal cases so the main path stays shallow.
- register `defer` cleanup immediately after acquiring a resource. use `errdefer` for cleanup needed only until ownership is transferred successfully.
## allocation and ownership
- put comptime parameters first, then the allocator as the first runtime parameter, then ordinary runtime inputs. for example: `duplicate(u32, allocator, source)`. use the same order for allocation, reallocation, and freeing operations.
- keep fixed-size temporary storage local. for dynamic storage, accept an allocator explicitly or use one retained by the object being operated on.
- an allocator argument means caller-controlled dynamic memory, not necessarily ownership of the result or an allocation on every call.
- document whether allocated storage is temporary, returned to the caller, or retained by an object. for caller-owned results, name the cleanup operation and any requirement to use the same allocator.
- release temporary storage before returning, including on error. objects that retain an allocator use it for subsequent allocations and release owned storage through their cleanup operation.
## documentation
- prefer lowercase prose in documentation, including headings and sentence starts. preserve the case of code identifiers, proper names, and acronyms.
- write comments for invariants, ownership, or reasons that the code alone does not explain. use `#!` for API documentation.
- enclose explicit references to parameter identifiers in backticks, for example: “an empty `destination` is a no-op.” leave general conceptual terms unquoted, even when they match a parameter name, for example: “memory must be freed with the same allocator that was used to allocate it.”
- when editing a function, review its existing comments and API documentation against the resulting implementation. update comments that remain useful; remove those that are obsolete or merely restate self-explanatory code. retain contracts and rationale that callers or maintainers cannot infer from the implementation.
## library operations
- reuse an existing standard-library operation when it expresses the work directly.
- use `mem.set(destination, value)` when assigning the same complete typed value to every element of a mutable slice. this preserves the value's fields; it is not a byte-wise zeroing operation.
- keep a loop when only one field is initialized, values differ by element, or initialization shares a pass with other work. a helper should not introduce unnecessary writes or another traversal.
when editing or reviewing code, apply every relevant guideline to the changed code. keep unrelated formatting changes separate.
+141 -23
View File
@@ -1,10 +1,11 @@
#! GRAMMAR:
#! grammar:
#! * type ..... -> identifier
#! * assignment -> :: | :=
#! * literal .. -> int | float | string
#! * expr ..... -> identifier | literal
#! * op ....... -> '+' | '-' | '*' | '/'
#! * expr ..... -> identifier | literal | '(' expr ')' | '-' expr | expr op expr | expr '(' [expr {',' expr} [',']] ')'
#! * decl ..... -> identifier [type] assignment expr
#! * stmt ..... -> decl
#! * stmt ..... -> decl | expr
import "@std/debug"
import "@std/mem"
@@ -13,20 +14,45 @@ import "@source/diagnostics"
Span :: alias diagnostics.Span
DiagnosticId :: alias diagnostics.DiagnosticId
NO_DIAGNOSTIC :: alias diagnostics.NO_DIAGNOSTIC
no_diagnostic :: alias diagnostics.no_diagnostic
NodeId :: distinct u32
@hide:file
NO_NODE :: maxval!(NodeId)
no_node :: maxval!(NodeId)
LiteralKind :: enum {
int, float, string
}
ExprKind :: enum {
literal_int
literal_float
literal_string
identifier
unary
binary
call
invalid
}
DeclKind :: enum {
statement, function, invalid
}
StmtKind :: enum {
declaration, expression, invalid
}
Mutability :: enum {
constant, variable
}
LiteralKind :: enum { int, float, string }
ExprKind :: enum { literal_int, literal_float, literal_string, identifier, unary, binary, invalid }
DeclKind :: enum { statement, function, invalid }
StmtKind :: enum { declaration, invalid }
Mutability :: enum { constant, variable }
UnaryOperator :: enum { negate }
BinaryOperator :: enum { add, subtract, multiply, divide }
BinaryOperator :: enum {
add, subtract, multiply, divide
}
@hide:file
NodeKind :: enum {
@@ -36,6 +62,8 @@ NodeKind :: enum {
expr_identifier,
expr_unary,
expr_binary,
expr_call,
stmt_expr,
stmt_decl,
func_decl,
param_decl,
@@ -59,9 +87,9 @@ NodeData :: union {
@hide:file
Node :: struct {
kind NodeKind
main NodeData = NodeData{ node_id = NO_NODE }
data0 NodeData = NodeData{ node_id = NO_NODE }
data1 NodeData = NodeData{ node_id = NO_NODE }
main NodeData = NodeData{ node_id = no_node }
data0 NodeData = NodeData{ node_id = no_node }
data1 NodeData = NodeData{ node_id = no_node }
}
@hide:file
@@ -71,11 +99,13 @@ Storage :: struct {
spans arraylist.ArrayList(Span)
}
#! owns node storage, retains its allocator, and borrows source until finish or deinit_builder.
Builder :: struct {
storage Storage
source []u8
}
#! owns node storage and borrows source; source must remain valid until deinit.
Tree :: struct {
storage Storage
source []u8
@@ -116,11 +146,17 @@ BinaryExprView :: struct {
right NodeId
}
CallExprView :: struct {
span Span
callee NodeId
}
InvalidView :: struct {
span Span
diagnostic DiagnosticId
}
#! retains `allocator` for node storage and borrows `source`; release with deinit_builder.
init_builder proc(allocator mem.Allocator, source []u8) Builder {
return Builder{
storage = Storage{
@@ -132,12 +168,14 @@ init_builder proc(allocator mem.Allocator, source []u8) Builder {
}
}
#! releases builder-owned node storage with its retained allocator, without freeing source.
deinit_builder proc(builder @mut Builder) void {
arraylist.deinit(&builder.storage.nodes)
arraylist.deinit(&builder.storage.extra)
arraylist.deinit(&builder.storage.spans)
}
#! transfers node storage and the source borrow to a caller-owned Tree; release with deinit.
finish proc(builder @mut Builder, root NodeId) Tree {
tree :: Tree{ storage = builder.storage, source = builder.source, root = root }
allocator :: builder.storage.nodes.allocator
@@ -150,6 +188,7 @@ finish proc(builder @mut Builder, root NodeId) Tree {
return tree
}
#! releases owned node storage with its retained allocator, without freeing source.
deinit proc(tree @mut Tree) void {
arraylist.deinit(&tree.storage.nodes)
arraylist.deinit(&tree.storage.extra)
@@ -197,6 +236,29 @@ add_binary proc(
}, span)
}
add_call proc(
builder @mut Builder,
span Span,
callee NodeId,
arguments []NodeId,
) NodeId ! mem.AllocError {
start :: builder.storage.extra.items.len
for (arguments) |argument| _ = try add_extra(builder, u32(argument))
return try add_node(builder, Node{
kind = .expr_call,
main = NodeData{ node_id = callee },
data0 = NodeData{ extra = u32(start) },
data1 = NodeData{ extra = u32(builder.storage.extra.items.len) },
}, span)
}
add_expression_stmt proc(builder @mut Builder, expression NodeId) NodeId ! mem.AllocError {
return try add_node(builder, Node{
kind = .stmt_expr,
main = NodeData{ node_id = expression },
}, builder_span(builder, expression))
}
add_statement_decl proc(
builder @mut Builder,
span Span,
@@ -205,7 +267,7 @@ add_statement_decl proc(
mutability Mutability,
initializer NodeId,
) NodeId ! mem.AllocError {
raw_type :: if (type) |id| u32(id) else u32(NO_NODE)
raw_type :: if (type) |id| u32(id) else u32(no_node)
extra :: try add_extra(builder, name.start)
_ = try add_extra(builder, name.end)
_ = try add_extra(builder, raw_type)
@@ -268,27 +330,43 @@ add_package proc(builder @mut Builder, span Span, declarations []NodeId) NodeId
}, span)
}
add_invalid_expr proc(builder @mut Builder, span Span, diagnostic DiagnosticId) NodeId ! mem.AllocError {
add_invalid_expr proc(
builder @mut Builder,
span Span,
diagnostic DiagnosticId,
) NodeId ! mem.AllocError {
return try add_node(builder, Node{
kind = .invalid_expr,
main = NodeData{ diagnostic = diagnostic },
}, span)
}
add_invalid_stmt proc(builder @mut Builder, span Span, diagnostic DiagnosticId) NodeId ! mem.AllocError {
add_invalid_stmt proc(
builder @mut Builder,
span Span,
diagnostic DiagnosticId,
) NodeId ! mem.AllocError {
return try add_node(builder, Node{
kind = .invalid_stmt,
main = NodeData{ diagnostic = diagnostic },
}, span)
}
add_invalid_decl proc(builder @mut Builder, span Span, diagnostic DiagnosticId) NodeId ! mem.AllocError {
add_invalid_decl proc(
builder @mut Builder,
span Span,
diagnostic DiagnosticId,
) NodeId ! mem.AllocError {
return try add_node(builder, Node{
kind = .invalid_decl,
main = NodeData{ diagnostic = diagnostic },
}, span)
}
node_count proc(tree @Tree) usize {
return tree.storage.nodes.items.len
}
package_decl_count proc(tree @Tree) usize {
root :: get_node(&tree.storage, tree.root)
debug.assert(root.kind == .package)
@@ -314,11 +392,18 @@ decl_kind proc(tree @Tree, id NodeId) DeclKind {
stmt_kind proc(tree @Tree, id NodeId) StmtKind {
return match get_node(&tree.storage, id).kind {
.stmt_decl: .declaration
.stmt_expr: .expression
.invalid_stmt: .invalid
else: unreachable
}
}
statement_expr proc(tree @Tree, id NodeId) NodeId {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .stmt_expr)
return node.main.node_id
}
statement_decl proc(tree @Tree, id NodeId) StatementDeclView {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .stmt_decl)
@@ -389,6 +474,7 @@ expr_kind proc(tree @Tree, id NodeId) ExprKind {
.expr_identifier: .identifier
.expr_unary: .unary
.expr_binary: .binary
.expr_call: .call
.invalid_expr: .invalid
else: unreachable
}
@@ -415,14 +501,46 @@ binary_expr proc(tree @Tree, id NodeId) BinaryExprView {
}
}
node_span proc(tree @Tree, id NodeId) Span { return get_span(&tree.storage, id) }
call_expr proc(tree @Tree, id NodeId) CallExprView {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .expr_call)
return CallExprView{
span = get_span(&tree.storage, id),
callee = node.main.node_id,
}
}
call_arg_count proc(tree @Tree, id NodeId) usize {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .expr_call)
return usize(node.data1.extra - node.data0.extra)
}
call_arg proc(tree @Tree, id NodeId, index usize) NodeId {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .expr_call)
debug.assert(index < usize(node.data1.extra - node.data0.extra))
return NodeId(tree.storage.extra.items[usize(node.data0.extra) + index])
}
node_span proc(tree @Tree, id NodeId) Span {
return get_span(&tree.storage, id)
}
invalid proc(tree @Tree, id NodeId) InvalidView {
node :: get_node(&tree.storage, id)
debug.assert(node.kind == .invalid_expr or node.kind == .invalid_stmt or node.kind == .invalid_decl)
return InvalidView{ span = get_span(&tree.storage, id), diagnostic = node.main.diagnostic }
debug.assert(
node.kind == .invalid_expr
or node.kind == .invalid_stmt
or node.kind == .invalid_decl
)
return InvalidView{
span = get_span(&tree.storage, id),
diagnostic = node.main.diagnostic,
}
}
#! returns text borrowed from the source buffer; the caller must not free it.
source_text proc(tree @Tree, span Span) []u8 {
debug.assert(span.start <= span.end and usize(span.end) <= tree.source.len)
return tree.source[usize(span.start)..usize(span.end)]
@@ -473,12 +591,12 @@ get_span proc(storage @Storage, id NodeId) Span {
@hide:file
node_id proc(index usize) NodeId {
debug.assert(u64(index) < u64(NO_NODE))
debug.assert(u64(index) < u64(no_node))
return NodeId(index)
}
@hide:file
maybe_node_id proc(value u32) ?NodeId {
if (value == u32(NO_NODE)) return null
if (value == u32(no_node)) return null
return NodeId(value)
}
+1 -1
View File
@@ -15,7 +15,7 @@ exposes_typed_declaration_views test {
.constant,
value,
)
declarations [1]NodeId := [declaration]
declarations [1]NodeId :: [declaration]
package :: try add_package(&builder, Span{ start = 0, end = 11 }, declarations[..])
tree := finish(&builder, package)
defer deinit(&tree)
+10
View File
@@ -2,18 +2,22 @@ import "@std/mem"
import "@std/arraylist"
import "@source/ast"
#! owns output bytes and retains its allocator; release with deinit.
Renderer :: struct {
bytes arraylist.ArrayList(u8)
}
#! retains `allocator` for rendered output; the caller owns the returned Renderer.
init proc(allocator mem.Allocator) Renderer {
return Renderer{ bytes = arraylist.init(allocator) }
}
#! releases output bytes with the retained allocator.
deinit proc(renderer @mut Renderer) void {
arraylist.deinit(&renderer.bytes)
}
#! borrows `tree` during rendering; output stays renderer-owned until render or deinit.
render proc(renderer @mut Renderer, tree @ast.Tree) void ! mem.AllocError {
arraylist.clear(&renderer.bytes)
try append_text(renderer, "package\n")
@@ -41,6 +45,12 @@ render_function proc(
statement :: ast.block_statement(tree, function.body, index)
match ast.stmt_kind(tree, statement) {
.declaration: try render_statement(renderer, tree, statement, " ")
.expression: {
expression :: ast.statement_expr(tree, statement)
try append_text(renderer, " expression ")
try append_text(renderer, ast.source_text(tree, ast.node_span(tree, expression)))
try append_text(renderer, "\n")
}
.invalid: try append_text(renderer, " invalid statement\n")
}
}
+1 -1
View File
@@ -14,7 +14,7 @@ renders_semantic_debug_view test {
.constant,
value,
)
declarations [1]ast.NodeId := [declaration]
declarations [1]ast.NodeId :: [declaration]
package :: try ast.add_package(&builder, ast.Span{ start = 0, end = 11 }, declarations[..])
tree := ast.finish(&builder, package)
defer ast.deinit(&tree)
+8 -2
View File
@@ -7,13 +7,15 @@ Span :: struct {
}
DiagnosticId :: distinct u32
NO_DIAGNOSTIC :: maxval!(DiagnosticId)
no_diagnostic :: maxval!(DiagnosticId)
Code :: enum {
invalid_character,
float_must_end_with_digit,
unterminated_string,
unexpected_token,
duplicate_name,
unknown_name,
}
Diagnostic :: struct {
@@ -21,20 +23,24 @@ Diagnostic :: struct {
span Span
}
#! owns diagnostic storage and retains its allocator; release with deinit.
Store :: struct { list arraylist.ArrayList(Diagnostic) }
CapacityError :: enum { diagnostic_limit_reached }
#! retains `allocator` for added diagnostics; the caller owns the returned Store.
init proc(allocator mem.Allocator) Store {
return Store{ list = arraylist.init(allocator) }
}
#! releases diagnostic storage with the retained allocator.
deinit proc(store @mut Store) void {
arraylist.deinit(&store.list)
}
#! allocates from the retained allocator; the returned ID is valid until deinit.
add proc(store @mut Store, code Code, span Span) DiagnosticId ! (mem.AllocError | CapacityError) {
if (store.list.items.len >= usize(NO_DIAGNOSTIC)) return .diagnostic_limit_reached
if (store.list.items.len >= usize(no_diagnostic)) return .diagnostic_limit_reached
id :: DiagnosticId(store.list.items.len)
try arraylist.append(&store.list, Diagnostic{ code = code, span = span })
return id
@@ -5,6 +5,7 @@ import "@std/arraylist"
import "@source/diagnostics"
import "@source/strpool"
@hide:file
ScanError :: struct {
code ErrorCode
end usize
@@ -17,7 +18,7 @@ ErrorCode :: enum {
}
SourceError :: enum { source_too_large }
@hide:file
keywords strmap.StaticStringMap(TokenKind) :: strmap.init([
{ "proc", .proc },
{ "return", .return },
@@ -27,18 +28,25 @@ keywords strmap.StaticStringMap(TokenKind) :: strmap.init([
{ "while", .while },
])
#! owns token storage and retains its allocator; release with deinit.
#! identifier IDs refer to strings in the separately owned strpool.strings.
State :: struct {
tokens arraylist.ArrayList(Token)
}
#! retains `allocator` for scanned tokens; the caller owns the returned State.
init proc(allocator mem.Allocator) State {
return State{ tokens = arraylist.init(allocator) }
}
#! releases token storage with the retained allocator, without releasing strpool.strings.
deinit proc(state @mut State) void {
arraylist.deinit(&state.tokens)
}
#! borrows `program` and `diagnostic_store` during scanning; tokens remain state-owned.
#! token allocation uses the retained allocator; identifiers use strpool.strings's allocator.
#! initialize strpool.strings before scanning and keep it alive while using identifier IDs.
scan proc(
state @mut State,
program []u8,
@@ -49,24 +57,22 @@ scan proc(
while cursor < program.len {
char :: program[cursor]
# whitespace
if char == '\n' {
try add_token(state, Token{ span = make_span(cursor, cursor + 1), kind = .newline })
cursor += 1
continue
} else if is_whitespace(char) {
}
if is_whitespace(char) {
cursor += 1
continue
}
# comments
if char == '#' {
while (cursor < program.len and program[cursor] != '\n') cursor += 1
if (cursor < program.len) cursor += 1
continue
}
# identifiers and keywords
if is_alpha(char) or char == '_' {
start :: cursor
result :: scan_ident(start, program)
@@ -79,7 +85,6 @@ scan proc(
continue
}
# numeric literals
if is_digit(char) {
start :: cursor
result :: scan_number(start, program) catch |err| {
@@ -102,7 +107,6 @@ scan proc(
continue
}
# string literals
if char == '"' {
start :: cursor
result :: scan_string(start, program) catch |err| {
@@ -124,15 +128,20 @@ scan proc(
continue
}
# immutable decl, mutable decl, or single colon
if char == ':' {
if cursor + 1 < program.len and program[cursor + 1] == ':' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .double_colon })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .double_colon,
})
cursor += 2
continue
}
if cursor + 1 < program.len and program[cursor + 1] == '=' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .colon_equal })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .colon_equal,
})
cursor += 2
continue
}
@@ -141,10 +150,12 @@ scan proc(
continue
}
# compound arithmetic assignment
if char == '+' {
if cursor + 1 < program.len and program[cursor + 1] == '=' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .plus_equal })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .plus_equal,
})
cursor += 2
continue
}
@@ -154,7 +165,10 @@ scan proc(
}
if char == '-' {
if cursor + 1 < program.len and program[cursor + 1] == '=' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .minus_equal })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .minus_equal,
})
cursor += 2
continue
}
@@ -164,7 +178,10 @@ scan proc(
}
if char == '*' {
if cursor + 1 < program.len and program[cursor + 1] == '=' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .star_equal })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .star_equal,
})
cursor += 2
continue
}
@@ -174,7 +191,10 @@ scan proc(
}
if char == '/' {
if cursor + 1 < program.len and program[cursor + 1] == '=' {
try add_token(state, Token{ span = make_span(cursor, cursor + 2), kind = .slash_equal })
try add_token(state, Token{
span = make_span(cursor, cursor + 2),
kind = .slash_equal,
})
cursor += 2
continue
}
@@ -183,7 +203,6 @@ scan proc(
continue
}
# single character tokens
kind ?TokenKind :: match char {
'=': .equal
',': .comma
@@ -201,7 +220,6 @@ scan proc(
continue
}
# invalid character
id :: try diagnostics.add(
diagnostic_store,
.invalid_character,
@@ -218,76 +236,86 @@ scan proc(
try add_token(state, Token{ span = make_span(cursor, cursor), kind = .eof })
}
@hide:file
ScanIdentResult :: struct {
end usize
kind TokenKind
str_id strpool.StringId
}
@hide:file
scan_ident proc(start usize, program []u8) ScanIdentResult {
cursor := start + 1
# scan whole identifier
while (cursor < program.len and (
is_alpha(program[cursor]) or
is_digit(program[cursor]) or
program[cursor] == '_'
)) cursor += 1
while (
cursor < program.len
and (
is_alpha(program[cursor])
or is_digit(program[cursor])
or program[cursor] == '_'
)
) {
cursor += 1
}
# don't intern keywords (already O(1) lookup via token kind)
kind :: strmap.get(&keywords, program[start..cursor]) orelse .ident
str_id :: if (kind == .ident)
strpool.intern(&strpool.STRINGS, program[start..cursor]) catch strpool.NO_STR
strpool.intern(&strpool.strings, program[start..cursor]) catch strpool.no_str
else
strpool.NO_STR
strpool.no_str
return ScanIdentResult{ end = cursor, kind = kind, str_id = str_id }
}
@hide:file
ScanNumResult :: struct {
end usize
has_decimal bool
}
@hide:file
scan_number proc(start usize, program []u8) ScanNumResult ! ScanError {
cursor := start
has_decimal := false
# scan integer part
while (cursor < program.len and is_digit(program[cursor])) cursor += 1
# check for decimal
if cursor < program.len and program[cursor] == '.' {
has_decimal = true
cursor += 1
}
# assert non-terminating decimal
if has_decimal and (cursor >= program.len or !is_digit(program[cursor])) {
return ScanError{ code = .float_must_end_with_digit, end = cursor }
return ScanError{
code = .float_must_end_with_digit,
end = cursor,
}
}
# scan fractional part
while (cursor < program.len and is_digit(program[cursor])) cursor += 1
return ScanNumResult{ end = cursor, has_decimal = has_decimal }
}
@hide:file
ScanStrResult :: struct { end usize }
@hide:file
scan_string proc(start usize, program []u8) ScanStrResult ! ScanError {
cursor := start + 1 # skip first `"`
# scan entire string
while cursor < program.len and program[cursor] != '"' and program[cursor] != '\n' : cursor += 1 {
# ignore escaped characters
# escaped quotes do not terminate the string.
if (program[cursor] == '\\' and cursor + 1 < program.len) cursor += 1
}
# assert string terminal
if (cursor >= program.len or program[cursor] != '"') {
return ScanError{ code = .unterminated_string, end = cursor }
return ScanError{
code = .unterminated_string,
end = cursor,
}
}
return ScanStrResult{ end = cursor + 1 } # skip last `"`
}
@hide
@hide:file
is_whitespace proc(char u8) bool {
return char == ' ' or char == '\t' or char == '\n' or char == '\r'
}
@hide
@hide:file
is_alpha proc(char u8) bool {
return match char {
'a'..='z', 'A'..='Z': true
@@ -295,7 +323,7 @@ is_alpha proc(char u8) bool {
}
}
@hide
@hide:file
is_digit proc(char u8) bool {
return match char {
'0'..='9': true
@@ -316,7 +344,7 @@ make_span proc(start, end usize) diagnostics.Span {
return diagnostics.Span{ start = u32(start), end = u32(end) }
}
@hide
@hide:file
add_token proc(state @mut State, token Token) void ! mem.AllocError {
_ = token_id(state.tokens.items.len)
try arraylist.append(&state.tokens, token)
@@ -4,8 +4,8 @@ import "@source/diagnostics"
import "@source/strpool"
reports_invalid_input_and_keeps_scanning test {
strpool.STRINGS = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.STRINGS)
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
state := init(mem.c_allocator)
@@ -23,7 +23,10 @@ reports_invalid_input_and_keeps_scanning test {
try testing.expect_equal(TokenKind.string, state.tokens.items[6].kind)
try testing.expect_equal(TokenKind.eof, state.tokens.items[7].kind)
try testing.expect_equal(2, diagnostics.count(&diagnostic_store))
try testing.expect_equal("name", strpool.get_str(&strpool.STRINGS, state.tokens.items[1].data.str_id)?)
try testing.expect_equal(
"name",
strpool.get_str(&strpool.strings, state.tokens.items[1].data.str_id)?,
)
try testing.expect_equal(u32(3), state.tokens.items[1].span.start)
try testing.expect_equal(u32(7), state.tokens.items[1].span.end)
}
+12 -12
View File
@@ -3,7 +3,7 @@ import "@source/diagnostics"
import "@source/strpool"
TokenId :: distinct u32
NO_TOKEN :: maxval!(TokenId)
no_token :: maxval!(TokenId)
TokenData :: union {
str_id strpool.StringId
@@ -13,7 +13,7 @@ TokenData :: union {
Token :: struct {
span diagnostics.Span
kind TokenKind
data TokenData = TokenData{ str_id = strpool.NO_STR }
data TokenData = TokenData{ str_id = strpool.no_str }
}
TokenKind :: enum {
@@ -57,22 +57,22 @@ TokenKind :: enum {
invalid
}
render_token proc(tok Token, program []u8) void {
match tok.kind {
render_token proc(token Token, program []u8) void {
match token.kind {
.ident: {
if (strpool.get_str(&strpool.STRINGS, tok.data.str_id)) |str| {
debug.print("{}({})\n", {tok.kind, str})
if (strpool.get_str(&strpool.strings, token.data.str_id)) |str| {
debug.print("{}({})\n", {token.kind, str})
} else {
debug.print("{}({})\n", {tok.kind, program[tok.span.start..tok.span.end]})
debug.print("{}({})\n", {token.kind, program[token.span.start..token.span.end]})
}
}
.string, .int, .float:
debug.print("{}({})\n", {tok.kind, program[tok.span.start..tok.span.end]})
else: debug.print("{}\n", {tok.kind})
debug.print("{}({})\n", {token.kind, program[token.span.start..token.span.end]})
else: debug.print("{}\n", {token.kind})
}
}
token_id proc(idx uint) TokenId {
debug.assert(u64(idx) < u64(NO_TOKEN))
return TokenId(idx)
token_id proc(index uint) TokenId {
debug.assert(u64(index) < u64(no_token))
return TokenId(index)
}
+16 -6
View File
@@ -6,9 +6,11 @@ import "@source/strpool"
import "@source/lexer"
import "@source/ast"
import "@source/parser"
import "@source/sema"
ast_renderer :: import "@source/ast/renderer"
test import "@std/mem"
test import "@std/enummap"
test import "@std/arraylist"
test import "@std/hashmap"
@@ -20,7 +22,9 @@ test import "@source/lexer"
test import "@source/ast"
test import "@source/ast/renderer"
test import "@source/parser"
test import "@source/sema"
@hide:file
program ::
`# literals
`x float :: 123.9
@@ -30,14 +34,17 @@ program ::
`a := 1 + 2 * 3
`b :: -3 / 2 + 1
`
`empty proc() void {}
`grouped proc(a, b T, c U) void {
` value int :: 1
`grouped proc(left, right int, extra float) void {
` value int :: later + left
` later int :: right
` empty()
` grouped(value, later, extra)
`}
`empty proc() void {}
main proc() void! {
strpool.STRINGS = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.STRINGS)
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
@@ -47,13 +54,16 @@ main proc() void! {
try lexer.scan(&scan_state, program, &diagnostic_store)
tree := try parser.parse(
mem.c_allocator,
scan_state.tokens.items,
program,
&diagnostic_store,
mem.c_allocator,
)
defer ast.deinit(&tree)
resolution := try sema.resolve(mem.c_allocator, &tree, &diagnostic_store)
defer sema.deinit(&resolution)
renderer := ast_renderer.init(mem.c_allocator)
defer ast_renderer.deinit(&renderer)
try ast_renderer.render(&renderer, &tree)
@@ -29,11 +29,14 @@ State :: struct {
next_token usize
}
#! borrows `tokens` and `source` while parsing; the returned Tree borrows `source` until ast.deinit.
#! `allocator` supplies returned node storage and temporary lists released before return.
#! the caller owns the Tree and must release it with ast.deinit.
parse proc(
allocator mem.Allocator,
tokens []lexer.Token,
source []u8,
diagnostic_store @mut diagnostics.Store,
allocator mem.Allocator,
) ast.Tree ! (mem.AllocError | diagnostics.CapacityError) {
state := State{
builder = ast.init_builder(allocator, source),
@@ -55,7 +58,7 @@ parse_package proc(state @mut State) ast.NodeId ! (mem.AllocError | diagnostics.
skip_newlines(state)
while !next_is(state, .eof) {
start :: state.next_token
declaration := parse_decl(state) catch |err| {
declaration :: parse_decl(state) catch |err| {
match err {
.unexpected_token: {
recovered :: try recover_decl(state, start)
@@ -79,9 +82,7 @@ parse_package proc(state @mut State) ast.NodeId ! (mem.AllocError | diagnostics.
@hide:file
parse_decl proc(state @mut State) ast.NodeId ! ParseError {
if next_is(state, .ident) and next_n_is(state, 1, .proc) {
return try parse_func_decl(state)
}
if (next_is(state, .ident) and next_n_is(state, 1, .proc)) return try parse_func_decl(state)
return try parse_statement_decl(state)
}
@@ -142,7 +143,7 @@ parse_block proc(state @mut State) ast.NodeId ! ParseError {
skip_newlines(state)
while !next_is(state, .close_curly) and !next_is(state, .eof) {
start :: state.next_token
statement := parse_statement(state) catch |err| {
statement :: parse_statement(state) catch |err| {
match err {
.unexpected_token: {
recovered :: try recover_statement(state, start)
@@ -167,14 +168,29 @@ parse_block proc(state @mut State) ast.NodeId ! ParseError {
@hide:file
parse_statement proc(state @mut State) ast.NodeId ! ParseError {
if next_is(state, .ident) return try parse_statement_decl(state)
return .unexpected_token
if (
next_is(state, .ident)
and (
next_n_is(state, 1, .ident)
or next_n_is(state, 1, .double_colon)
or next_n_is(state, 1, .colon_equal)
)
) {
return try parse_statement_decl(state)
}
match current(state).kind {
.ident, .int, .float, .string, .minus, .open_paren: {
expression :: try parse_expr(state)
return try ast.add_expression_stmt(&state.builder, expression)
}
else: return .unexpected_token
}
}
@hide:file
parse_statement_decl proc(state @mut State) ast.NodeId ! ParseError {
name :: try consume(state, .ident)
type_id ?ast.NodeId := if next_is(state, .ident)
type_id ?ast.NodeId :: if next_is(state, .ident)
try parse_type(state)
else
null
@@ -202,12 +218,12 @@ parse_type proc(state @mut State) ast.NodeId ! ParseError {
}
@hide:file
parse_expr proc(state @mut State) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
parse_expr proc(state @mut State) ast.NodeId ! ParseError {
return try parse_expr_bp(state, BindingPowerScalar(0))
}
@hide:file
parse_expr_bp proc(state @mut State, min_bp BindingPowerScalar) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
parse_expr_bp proc(state @mut State, min_bp BindingPowerScalar) ast.NodeId ! ParseError {
token := current(state)
left := if (prefix_binding_power(token.kind)) |right_bp| {
state.next_token += 1
@@ -225,6 +241,10 @@ parse_expr_bp proc(state @mut State, min_bp BindingPowerScalar) ast.NodeId ! (me
while true {
token = current(state)
if (token.kind == .open_paren) {
left = try parse_call(state, left)
continue
}
if (infix_binding_power(token.kind)) |power| {
if (power.left < min_bp) break
state.next_token += 1
@@ -246,9 +266,55 @@ parse_expr_bp proc(state @mut State, min_bp BindingPowerScalar) ast.NodeId ! (me
}
@hide:file
parse_primary proc(state @mut State) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
parse_call proc(state @mut State, callee ast.NodeId) ast.NodeId ! ParseError {
_ = try consume(state, .open_paren)
arguments arraylist.ArrayList(ast.NodeId) := arraylist.init(state.allocator)
defer arraylist.deinit(&arguments)
while (
!next_is(state, .close_paren)
and !next_is(state, .newline)
and !next_is(state, .close_curly)
and !next_is(state, .eof)
) {
argument :: try parse_expr(state)
try arraylist.append(&arguments, argument)
if (!next_is(state, .comma)) break
state.next_token += 1
}
close :: try consume(state, .close_paren)
return try ast.add_call(
&state.builder,
ast.Span{
start = ast.builder_span(&state.builder, callee).start,
end = close.span.end,
},
callee,
arguments.items,
)
}
@hide:file
parse_primary proc(state @mut State) ast.NodeId ! ParseError {
token :: current(state)
match token.kind {
.open_paren: {
state.next_token += 1
expression :: try parse_expr(state)
if next_is(state, .close_paren) {
state.next_token += 1
return expression
}
diagnostic :: try diagnostics.add(
state.diagnostics,
.unexpected_token,
current(state).span,
)
return try ast.add_invalid_expr(&state.builder, ast.Span{
start = token.span.start,
end = ast.builder_span(&state.builder, expression).end,
}, diagnostic)
}
.int: {
state.next_token += 1
return try ast.add_literal(&state.builder, .int, token.span)
@@ -266,27 +332,41 @@ parse_primary proc(state @mut State) ast.NodeId ! (mem.AllocError | diagnostics.
return try ast.add_identifier(&state.builder, token.span)
}
else: {
diagnostic :: if (token.data.diagnostic != diagnostics.NO_DIAGNOSTIC)
diagnostic :: if (token.data.diagnostic != diagnostics.no_diagnostic)
token.data.diagnostic
else
try diagnostics.add(state.diagnostics, .unexpected_token, token.span)
if (token.kind != .newline and token.kind != .close_curly and token.kind != .eof) {
if (
token.kind != .newline
and token.kind != .comma
and token.kind != .close_paren
and token.kind != .close_curly
and token.kind != .eof
) {
state.next_token += 1
}
return try ast.add_invalid_expr(&state.builder, token.span, diagnostic)
}
}
}
@hide:file
recover_decl proc(state @mut State, start usize) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
recover_decl proc(
state @mut State,
start usize,
) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
diagnostic :: try diagnostics.add(state.diagnostics, .unexpected_token, current(state).span)
span :: synchronize(state, start, false)
return try ast.add_invalid_decl(&state.builder, span, diagnostic)
}
@hide:file
recover_statement proc(state @mut State, start usize) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
recover_statement proc(
state @mut State,
start usize,
) ast.NodeId ! (mem.AllocError | diagnostics.CapacityError) {
diagnostic :: try diagnostics.add(state.diagnostics, .unexpected_token, current(state).span)
span :: synchronize(state, start, true)
return try ast.add_invalid_stmt(&state.builder, span, diagnostic)
+287
View File
@@ -0,0 +1,287 @@
import "@std/mem"
import "@std/testing"
import "@source/diagnostics"
import "@source/strpool"
import "@source/lexer"
import "@source/ast"
parses_expression_precedence_through_tree_views test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source :: "value :: 1 + 2 * 3\n"
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(1, ast.package_decl_count(&tree))
declaration :: ast.package_decl(&tree, 0)
statement :: ast.statement_decl(&tree, declaration)
try testing.expect_equal("value", ast.source_text(&tree, statement.name))
try testing.expect_equal(ast.ExprKind.binary, ast.expr_kind(&tree, statement.initializer))
add :: ast.binary_expr(&tree, statement.initializer)
try testing.expect_equal(ast.BinaryOperator.add, add.operator)
try testing.expect_equal(ast.ExprKind.literal_int, ast.expr_kind(&tree, add.left))
try testing.expect_equal(ast.ExprKind.binary, ast.expr_kind(&tree, add.right))
multiply :: ast.binary_expr(&tree, add.right)
try testing.expect_equal(ast.BinaryOperator.multiply, multiply.operator)
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
}
parses_grouped_expression_precedence test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source :: "value :: a + ((b - c)) * 2 / 3\n"
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
try testing.expect_equal(1, ast.package_decl_count(&tree))
statement :: ast.statement_decl(&tree, ast.package_decl(&tree, 0))
add :: ast.binary_expr(&tree, statement.initializer)
try testing.expect_equal(ast.BinaryOperator.add, add.operator)
try testing.expect_equal("a", ast.source_text(&tree, ast.node_span(&tree, add.left)))
divide :: ast.binary_expr(&tree, add.right)
try testing.expect_equal(ast.BinaryOperator.divide, divide.operator)
try testing.expect_equal("3", ast.source_text(&tree, ast.node_span(&tree, divide.right)))
multiply :: ast.binary_expr(&tree, divide.left)
try testing.expect_equal(ast.BinaryOperator.multiply, multiply.operator)
try testing.expect_equal("2", ast.source_text(&tree, ast.node_span(&tree, multiply.right)))
subtract :: ast.binary_expr(&tree, multiply.left)
try testing.expect_equal(ast.BinaryOperator.subtract, subtract.operator)
try testing.expect_equal("b", ast.source_text(&tree, ast.node_span(&tree, subtract.left)))
try testing.expect_equal("c", ast.source_text(&tree, ast.node_span(&tree, subtract.right)))
}
parses_function_structure_through_tree_views test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`grouped proc(a, b T, c U) void {
` value int :: 1
`}
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
declaration :: ast.package_decl(&tree, 0)
try testing.expect_equal(ast.DeclKind.function, ast.decl_kind(&tree, declaration))
function :: ast.function_decl(&tree, declaration)
try testing.expect_equal("grouped", ast.source_text(&tree, function.name))
try testing.expect_equal(3, ast.function_param_count(&tree, declaration))
first :: ast.param(&tree, ast.function_param(&tree, declaration, 0))
second :: ast.param(&tree, ast.function_param(&tree, declaration, 1))
third :: ast.param(&tree, ast.function_param(&tree, declaration, 2))
try testing.expect_equal("a", ast.source_text(&tree, first.name))
try testing.expect_equal("b", ast.source_text(&tree, second.name))
try testing.expect_equal("c", ast.source_text(&tree, third.name))
try testing.expect_equal(first.type, second.type)
try testing.expect(first.type != third.type)
try testing.expect_equal(1, ast.block_statement_count(&tree, function.body))
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
}
recovers_invalid_declaration_and_preserves_sibling test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source :: "@ broken\nvalid :: 1\n"
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(2, ast.package_decl_count(&tree))
invalid_decl :: ast.package_decl(&tree, 0)
valid_decl :: ast.package_decl(&tree, 1)
try testing.expect_equal(ast.DeclKind.invalid, ast.decl_kind(&tree, invalid_decl))
try testing.expect_equal(ast.DeclKind.statement, ast.decl_kind(&tree, valid_decl))
invalid_view :: ast.invalid(&tree, invalid_decl)
try testing.expect(diagnostics.get(&diagnostic_store, invalid_view.diagnostic) != null)
valid_statement :: ast.statement_decl(&tree, valid_decl)
try testing.expect_equal("valid", ast.source_text(&tree, valid_statement.name))
}
recovers_malformed_groups_and_preserves_following_statements test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`main proc() void {
` empty :: ()
` operand :: (1 + )
` missing :: (2
` valid :: (3)
`}
`after :: 4
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(3, diagnostics.count(&diagnostic_store))
try testing.expect_equal(2, ast.package_decl_count(&tree))
function :: ast.function_decl(&tree, ast.package_decl(&tree, 0))
try testing.expect_equal(4, ast.block_statement_count(&tree, function.body))
empty :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 0))
try testing.expect_equal(ast.ExprKind.invalid, ast.expr_kind(&tree, empty.initializer))
operand :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 1))
add :: ast.binary_expr(&tree, operand.initializer)
try testing.expect_equal(ast.ExprKind.invalid, ast.expr_kind(&tree, add.right))
missing :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 2))
try testing.expect_equal(ast.ExprKind.invalid, ast.expr_kind(&tree, missing.initializer))
valid :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 3))
try testing.expect_equal(ast.ExprKind.literal_int, ast.expr_kind(&tree, valid.initializer))
try testing.expect_equal("3", ast.source_text(&tree, ast.node_span(&tree, valid.initializer)))
after :: ast.statement_decl(&tree, ast.package_decl(&tree, 1))
try testing.expect_equal("after", ast.source_text(&tree, after.name))
try testing.expect_equal("4", ast.source_text(&tree, ast.node_span(&tree, after.initializer)))
}
parses_call_statements_and_postfix_precedence test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`main proc() void {
` ping()
` result int := -factory()(add(1, 2 * 3), (4 + 5),) + 6
` (handler)(result)
`}
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
function :: ast.function_decl(&tree, ast.package_decl(&tree, 0))
try testing.expect_equal(3, ast.block_statement_count(&tree, function.body))
ping_stmt :: ast.block_statement(&tree, function.body, 0)
try testing.expect_equal(ast.StmtKind.expression, ast.stmt_kind(&tree, ping_stmt))
ping :: ast.statement_expr(&tree, ping_stmt)
try testing.expect_equal(ast.ExprKind.call, ast.expr_kind(&tree, ping))
try testing.expect_equal(0, ast.call_arg_count(&tree, ping))
try testing.expect_equal(
"ping",
ast.source_text(&tree, ast.node_span(&tree, ast.call_expr(&tree, ping).callee)),
)
try testing.expect_equal("ping()", ast.source_text(&tree, ast.call_expr(&tree, ping).span))
result :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 1))
try testing.expect_equal(ast.Mutability.variable, result.mutability)
add :: ast.binary_expr(&tree, result.initializer)
try testing.expect_equal(ast.BinaryOperator.add, add.operator)
try testing.expect_equal("6", ast.source_text(&tree, ast.node_span(&tree, add.right)))
negate :: ast.unary_expr(&tree, add.left)
call :: negate.operand
try testing.expect_equal(2, ast.call_arg_count(&tree, call))
factory :: ast.call_expr(&tree, call).callee
try testing.expect_equal(0, ast.call_arg_count(&tree, factory))
try testing.expect_equal(
"factory",
ast.source_text(&tree, ast.node_span(&tree, ast.call_expr(&tree, factory).callee)),
)
nested :: ast.call_arg(&tree, call, 0)
try testing.expect_equal(2, ast.call_arg_count(&tree, nested))
try testing.expect_equal(
"add",
ast.source_text(&tree, ast.node_span(&tree, ast.call_expr(&tree, nested).callee)),
)
try testing.expect_equal(
"1",
ast.source_text(&tree, ast.node_span(&tree, ast.call_arg(&tree, nested, 0))),
)
multiply :: ast.binary_expr(&tree, ast.call_arg(&tree, nested, 1))
try testing.expect_equal(ast.BinaryOperator.multiply, multiply.operator)
try testing.expect_equal("2", ast.source_text(&tree, ast.node_span(&tree, multiply.left)))
try testing.expect_equal("3", ast.source_text(&tree, ast.node_span(&tree, multiply.right)))
grouped :: ast.binary_expr(&tree, ast.call_arg(&tree, call, 1))
try testing.expect_equal(ast.BinaryOperator.add, grouped.operator)
try testing.expect_equal("4", ast.source_text(&tree, ast.node_span(&tree, grouped.left)))
try testing.expect_equal("5", ast.source_text(&tree, ast.node_span(&tree, grouped.right)))
handler :: ast.statement_expr(&tree, ast.block_statement(&tree, function.body, 2))
try testing.expect_equal(
"handler",
ast.source_text(&tree, ast.node_span(&tree, ast.call_expr(&tree, handler).callee)),
)
try testing.expect_equal(1, ast.call_arg_count(&tree, handler))
try testing.expect_equal(
"result",
ast.source_text(&tree, ast.node_span(&tree, ast.call_arg(&tree, handler, 0))),
)
}
recovers_malformed_calls_and_preserves_siblings test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`main proc() void {
` bad(1 2)
` good()
` missing(1
` after :: 3
` holes(, 2)
`}
`tail :: f()
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(3, diagnostics.count(&diagnostic_store))
try testing.expect_equal(2, ast.package_decl_count(&tree))
function :: ast.function_decl(&tree, ast.package_decl(&tree, 0))
try testing.expect_equal(5, ast.block_statement_count(&tree, function.body))
for ([0, 2]) |index| {
statement :: ast.block_statement(&tree, function.body, usize(index))
try testing.expect_equal(ast.StmtKind.invalid, ast.stmt_kind(&tree, statement))
try testing.expect(
diagnostics.get(&diagnostic_store, ast.invalid(&tree, statement).diagnostic) != null,
)
}
good :: ast.statement_expr(&tree, ast.block_statement(&tree, function.body, 1))
try testing.expect_equal(
"good",
ast.source_text(&tree, ast.node_span(&tree, ast.call_expr(&tree, good).callee)),
)
after :: ast.statement_decl(&tree, ast.block_statement(&tree, function.body, 3))
try testing.expect_equal("after", ast.source_text(&tree, after.name))
holes :: ast.statement_expr(&tree, ast.block_statement(&tree, function.body, 4))
try testing.expect_equal(2, ast.call_arg_count(&tree, holes))
try testing.expect_equal(ast.ExprKind.invalid, ast.expr_kind(&tree, ast.call_arg(&tree, holes, 0)))
try testing.expect_equal(
"2",
ast.source_text(&tree, ast.node_span(&tree, ast.call_arg(&tree, holes, 1))),
)
tail :: ast.statement_decl(&tree, ast.package_decl(&tree, 1))
try testing.expect_equal("tail", ast.source_text(&tree, tail.name))
try testing.expect_equal(0, ast.call_arg_count(&tree, tail.initializer))
}
-91
View File
@@ -1,91 +0,0 @@
import "@std/mem"
import "@std/testing"
import "@source/diagnostics"
import "@source/strpool"
import "@source/lexer"
import "@source/ast"
parses_expression_precedence_through_tree_views test {
strpool.STRINGS = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.STRINGS)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source :: "value :: 1 + 2 * 3\n"
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(scan_state.tokens.items, source, &diagnostic_store, mem.c_allocator)
defer ast.deinit(&tree)
try testing.expect_equal(1, ast.package_decl_count(&tree))
declaration :: ast.package_decl(&tree, 0)
statement :: ast.statement_decl(&tree, declaration)
try testing.expect_equal("value", ast.source_text(&tree, statement.name))
try testing.expect_equal(ast.ExprKind.binary, ast.expr_kind(&tree, statement.initializer))
add :: ast.binary_expr(&tree, statement.initializer)
try testing.expect_equal(ast.BinaryOperator.add, add.operator)
try testing.expect_equal(ast.ExprKind.literal_int, ast.expr_kind(&tree, add.left))
try testing.expect_equal(ast.ExprKind.binary, ast.expr_kind(&tree, add.right))
multiply :: ast.binary_expr(&tree, add.right)
try testing.expect_equal(ast.BinaryOperator.multiply, multiply.operator)
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
}
parses_function_structure_through_tree_views test {
strpool.STRINGS = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.STRINGS)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`grouped proc(a, b T, c U) void {
` value int :: 1
`}
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(scan_state.tokens.items, source, &diagnostic_store, mem.c_allocator)
defer ast.deinit(&tree)
declaration :: ast.package_decl(&tree, 0)
try testing.expect_equal(ast.DeclKind.function, ast.decl_kind(&tree, declaration))
function :: ast.function_decl(&tree, declaration)
try testing.expect_equal("grouped", ast.source_text(&tree, function.name))
try testing.expect_equal(3, ast.function_param_count(&tree, declaration))
first :: ast.param(&tree, ast.function_param(&tree, declaration, 0))
second :: ast.param(&tree, ast.function_param(&tree, declaration, 1))
third :: ast.param(&tree, ast.function_param(&tree, declaration, 2))
try testing.expect_equal("a", ast.source_text(&tree, first.name))
try testing.expect_equal("b", ast.source_text(&tree, second.name))
try testing.expect_equal("c", ast.source_text(&tree, third.name))
try testing.expect_equal(first.type, second.type)
try testing.expect(first.type != third.type)
try testing.expect_equal(1, ast.block_statement_count(&tree, function.body))
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
}
recovers_invalid_declaration_and_preserves_sibling test {
strpool.STRINGS = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.STRINGS)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source :: "@ broken\nvalid :: 1\n"
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parse(scan_state.tokens.items, source, &diagnostic_store, mem.c_allocator)
defer ast.deinit(&tree)
try testing.expect_equal(2, ast.package_decl_count(&tree))
invalid_decl :: ast.package_decl(&tree, 0)
valid_decl :: ast.package_decl(&tree, 1)
try testing.expect_equal(ast.DeclKind.invalid, ast.decl_kind(&tree, invalid_decl))
try testing.expect_equal(ast.DeclKind.statement, ast.decl_kind(&tree, valid_decl))
invalid_view :: ast.invalid(&tree, invalid_decl)
try testing.expect(diagnostics.get(&diagnostic_store, invalid_view.diagnostic) != null)
valid_statement :: ast.statement_decl(&tree, valid_decl)
try testing.expect_equal("valid", ast.source_text(&tree, valid_statement.name))
}
+253
View File
@@ -0,0 +1,253 @@
import "@std/debug"
import "@std/mem"
import "@std/hashmap"
import "@std/strmap"
import "@source/ast"
import "@source/diagnostics"
BuiltinType :: enum { int, float, string, void }
@hide:file
builtins strmap.StaticStringMap(BuiltinType) :: strmap.init([
{ "int", .int },
{ "float", .float },
{ "string", .string },
{ "void", .void },
])
BindingKind :: enum { none, declaration, builtin, invalid }
BindingData :: union {
declaration ast.NodeId
builtin BuiltinType
diagnostic diagnostics.DiagnosticId
}
Binding :: struct {
kind BindingKind = .none
data BindingData = BindingData{ diagnostic = diagnostics.no_diagnostic }
}
@hide:file
Storage :: struct {
bindings []mut Binding
allocator mem.Allocator
}
#! owns binding storage and retains its allocator; release with deinit.
#! node and diagnostic IDs refer to the caller's Tree and diagnostics.Store.
Resolution :: struct { storage Storage }
@hide:file
Scope :: struct {
names hashmap.StringHashMap(Binding)
parent ?@Scope
}
@hide:file
State :: struct {
tree @ast.Tree
diagnostics @mut diagnostics.Store
bindings []mut Binding
allocator mem.Allocator
}
#! borrows `tree` and `diagnostic_store` during resolution, without retaining their storage.
#! `allocator` supplies caller-owned bindings and temporary scopes released before return.
#! release the returned Resolution with deinit, which uses the retained allocator.
resolve proc(
allocator mem.Allocator,
tree @ast.Tree,
diagnostic_store @mut diagnostics.Store,
) Resolution ! (mem.AllocError | diagnostics.CapacityError) {
bindings :: try mem.alloc(Binding, allocator, ast.node_count(tree))
errdefer mem.free(allocator, bindings)
mem.set(bindings, Binding{})
state := State{
tree = tree,
diagnostics = diagnostic_store,
bindings = bindings,
allocator = allocator,
}
package_scope := Scope{ names = hashmap.init(allocator), parent = null }
defer hashmap.deinit(&package_scope.names)
for (0..ast.package_decl_count(tree)) |i| {
id :: ast.package_decl(tree, i)
match ast.decl_kind(tree, id) {
.statement: try predeclare(&state, &package_scope, id, ast.statement_decl(tree, id).name)
.function: try predeclare(&state, &package_scope, id, ast.function_decl(tree, id).name)
.invalid: {}
}
}
for (0..ast.package_decl_count(tree)) |i| {
id :: ast.package_decl(tree, i)
match ast.decl_kind(tree, id) {
.statement: try resolve_statement_decl(&state, &package_scope, id)
.function: try resolve_function(&state, &package_scope, id)
.invalid: resolve_invalid(&state, id)
}
}
return Resolution{ storage = Storage{ bindings = bindings, allocator = allocator } }
}
#! releases binding storage with the allocator retained by resolve.
deinit proc(resolution @mut Resolution) void {
mem.free(resolution.storage.allocator, resolution.storage.bindings)
}
binding proc(resolution @Resolution, id ast.NodeId) Binding {
debug.assert(usize(id) < resolution.storage.bindings.len)
return resolution.storage.bindings[usize(id)]
}
@hide:file
lookup proc(scope ?@Scope, name []u8) ?Binding {
if scope |current| {
if hashmap.get(&current.names, name) |found| return found
return lookup(current.parent, name)
}
if strmap.get(&builtins, name) |type| {
return Binding{ kind = .builtin, data = BindingData{ builtin = type } }
}
return null
}
@hide:file
put_name proc(scope @mut Scope, name []u8, value Binding) void ! mem.AllocError {
hashmap.put(&scope.names, name, value) catch |err| {
match err {
.key_exists: unreachable
else: return err
}
}
}
@hide:file
predeclare proc(
state @mut State,
scope @mut Scope,
id ast.NodeId,
name_span ast.Span,
) void ! (mem.AllocError | diagnostics.CapacityError) {
name :: ast.source_text(state.tree, name_span)
if hashmap.get(&scope.names, name) |_| {
diagnostic :: try diagnostics.add(state.diagnostics, .duplicate_name, name_span)
state.bindings[usize(id)] = Binding{
kind = .invalid,
data = BindingData{ diagnostic = diagnostic },
}
return
}
value := Binding{ kind = .declaration, data = BindingData{ declaration = id } }
if lookup(scope.parent, name) |_| {
diagnostic :: try diagnostics.add(state.diagnostics, .duplicate_name, name_span)
# poison the inner name rather than silently binding uses to its ancestor.
value = Binding{ kind = .invalid, data = BindingData{ diagnostic = diagnostic } }
}
try put_name(scope, name, value)
state.bindings[usize(id)] = value
}
@hide:file
resolve_function proc(
state @mut State,
package_scope @Scope,
id ast.NodeId,
) void ! (mem.AllocError | diagnostics.CapacityError) {
tree :: state.tree
function :: ast.function_decl(tree, id)
# signatures cannot see parameters or body-local declarations.
try resolve_expr(state, package_scope, function.return_type)
for (0..ast.function_param_count(tree, id)) |i| {
parameter :: ast.function_param(tree, id, i)
try resolve_expr(state, package_scope, ast.param(tree, parameter).type)
}
scope := Scope{ names = hashmap.init(state.allocator), parent = package_scope }
defer hashmap.deinit(&scope.names)
for (0..ast.function_param_count(tree, id)) |i| {
parameter :: ast.function_param(tree, id, i)
try predeclare(state, &scope, parameter, ast.param(tree, parameter).name)
}
for (0..ast.block_statement_count(tree, function.body)) |i| {
statement :: ast.block_statement(tree, function.body, i)
if ast.stmt_kind(tree, statement) == .declaration {
try predeclare(state, &scope, statement, ast.statement_decl(tree, statement).name)
}
}
for (0..ast.block_statement_count(tree, function.body)) |i| {
statement :: ast.block_statement(tree, function.body, i)
match ast.stmt_kind(tree, statement) {
.declaration: try resolve_statement_decl(state, &scope, statement)
.expression: try resolve_expr(state, &scope, ast.statement_expr(tree, statement))
.invalid: resolve_invalid(state, statement)
}
}
}
@hide:file
resolve_statement_decl proc(
state @mut State,
scope @Scope,
id ast.NodeId,
) void ! (mem.AllocError | diagnostics.CapacityError) {
declaration :: ast.statement_decl(state.tree, id)
if (declaration.type) |type| try resolve_expr(state, scope, type)
try resolve_expr(state, scope, declaration.initializer)
}
@hide:file
resolve_invalid proc(state @mut State, id ast.NodeId) void {
if (state.bindings[usize(id)].kind != .none) return
state.bindings[usize(id)] = Binding{
kind = .invalid,
data = BindingData{ diagnostic = ast.invalid(state.tree, id).diagnostic },
}
}
@hide:file
resolve_expr proc(
state @mut State,
scope @Scope,
id ast.NodeId,
) void ! (mem.AllocError | diagnostics.CapacityError) {
# grouped parameters share their type node, including any diagnostic it owns.
if (state.bindings[usize(id)].kind != .none) return
tree :: state.tree
match ast.expr_kind(tree, id) {
.literal_int, .literal_float, .literal_string: {}
.identifier: {
span :: ast.node_span(tree, id)
name :: ast.source_text(tree, span)
if lookup(scope, name) |found| {
state.bindings[usize(id)] = found
} else {
diagnostic :: try diagnostics.add(state.diagnostics, .unknown_name, span)
state.bindings[usize(id)] = Binding{
kind = .invalid,
data = BindingData{ diagnostic = diagnostic },
}
}
}
.unary: try resolve_expr(state, scope, ast.unary_expr(tree, id).operand)
.binary: {
expression :: ast.binary_expr(tree, id)
try resolve_expr(state, scope, expression.left)
try resolve_expr(state, scope, expression.right)
}
.call: {
try resolve_expr(state, scope, ast.call_expr(tree, id).callee)
for (0..ast.call_arg_count(tree, id)) |i| {
try resolve_expr(state, scope, ast.call_arg(tree, id, i))
}
}
.invalid: resolve_invalid(state, id)
}
}
+300
View File
@@ -0,0 +1,300 @@
import "@std/mem"
import "@std/testing"
import "@source/diagnostics"
import "@source/strpool"
import "@source/lexer"
import "@source/parser"
import "@source/ast"
resolves_forward_references_recursion_and_builtin_signatures test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`first :: later
`alpha proc(p int, q float, s string) void {
` before :: -after * p + beta(after)
` after :: later
`}
`beta proc(p int) void {
` after :: p
` alpha(after, 1.0, "x")
`}
`later int :: 1
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parser.parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
resolution := try resolve(mem.c_allocator, &tree, &diagnostic_store)
defer deinit(&resolution)
first :: ast.package_decl(&tree, 0)
alpha :: ast.package_decl(&tree, 1)
beta :: ast.package_decl(&tree, 2)
later :: ast.package_decl(&tree, 3)
alpha_view :: ast.function_decl(&tree, alpha)
beta_view :: ast.function_decl(&tree, beta)
before :: ast.block_statement(&tree, alpha_view.body, 0)
after :: ast.block_statement(&tree, alpha_view.body, 1)
p :: ast.function_param(&tree, alpha, 0)
q :: ast.function_param(&tree, alpha, 1)
s :: ast.function_param(&tree, alpha, 2)
sibling_p :: ast.function_param(&tree, beta, 0)
sibling_after :: ast.block_statement(&tree, beta_view.body, 0)
first_ref :: ast.statement_decl(&tree, first).initializer
after_ref :: ast.statement_decl(&tree, after).initializer
references [2]ast.NodeId :: [first_ref, after_ref]
for references |id| {
try testing.expect_equal(BindingKind.declaration, binding(&resolution, id).kind)
try testing.expect_equal(later, binding(&resolution, id).data.declaration)
}
add :: ast.binary_expr(&tree, ast.statement_decl(&tree, before).initializer)
multiply :: ast.binary_expr(&tree, add.left)
forward_ref :: ast.unary_expr(&tree, multiply.left).operand
try testing.expect_equal(BindingKind.declaration, binding(&resolution, forward_ref).kind)
try testing.expect_equal(after, binding(&resolution, forward_ref).data.declaration)
try testing.expect_equal(BindingKind.declaration, binding(&resolution, multiply.right).kind)
try testing.expect_equal(p, binding(&resolution, multiply.right).data.declaration)
beta_ref :: ast.call_expr(&tree, add.right).callee
try testing.expect_equal(BindingKind.declaration, binding(&resolution, beta_ref).kind)
try testing.expect_equal(beta, binding(&resolution, beta_ref).data.declaration)
argument :: ast.call_arg(&tree, add.right, 0)
try testing.expect_equal(BindingKind.declaration, binding(&resolution, argument).kind)
try testing.expect_equal(after, binding(&resolution, argument).data.declaration)
recursive_call :: ast.statement_expr(&tree, ast.block_statement(&tree, beta_view.body, 1))
alpha_ref :: ast.call_expr(&tree, recursive_call).callee
try testing.expect_equal(BindingKind.declaration, binding(&resolution, alpha_ref).kind)
try testing.expect_equal(alpha, binding(&resolution, alpha_ref).data.declaration)
sibling_ref :: ast.call_arg(&tree, recursive_call, 0)
try testing.expect_equal(BindingKind.declaration, binding(&resolution, sibling_ref).kind)
try testing.expect_equal(sibling_after, binding(&resolution, sibling_ref).data.declaration)
sibling_param_ref :: ast.statement_decl(&tree, sibling_after).initializer
try testing.expect_equal(BindingKind.declaration, binding(&resolution, sibling_param_ref).kind)
try testing.expect_equal(sibling_p, binding(&resolution, sibling_param_ref).data.declaration)
parameter_types [2]ast.NodeId :: [ast.param(&tree, p).type, ast.param(&tree, sibling_p).type]
for parameter_types |id| {
try testing.expect_equal(BindingKind.builtin, binding(&resolution, id).kind)
try testing.expect_equal(BuiltinType.int, binding(&resolution, id).data.builtin)
}
if (ast.statement_decl(&tree, later).type) |type| {
try testing.expect_equal(BindingKind.builtin, binding(&resolution, type).kind)
try testing.expect_equal(BuiltinType.int, binding(&resolution, type).data.builtin)
} else {
return .expectation_failed
}
try testing.expect_equal(
BindingKind.builtin,
binding(&resolution, ast.param(&tree, q).type).kind,
)
try testing.expect_equal(
BuiltinType.float,
binding(&resolution, ast.param(&tree, q).type).data.builtin,
)
try testing.expect_equal(
BindingKind.builtin,
binding(&resolution, ast.param(&tree, s).type).kind,
)
try testing.expect_equal(
BuiltinType.string,
binding(&resolution, ast.param(&tree, s).type).data.builtin,
)
return_types [2]ast.NodeId :: [alpha_view.return_type, beta_view.return_type]
for return_types |id| {
try testing.expect_equal(BindingKind.builtin, binding(&resolution, id).kind)
try testing.expect_equal(BuiltinType.void, binding(&resolution, id).data.builtin)
}
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
}
rejects_shadowing_and_duplicates_without_cascades test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`collide proc(outer int, p int, p int) outer {
` outer :: missing
` outer :: 1
` p :: 1
` int :: 1
` pkg :: 1
` outer
` p
` int
` pkg
`}
`outer :: 1
`pkg :: 1
`same :: 1
`same :: same
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parser.parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
try testing.expect_equal(0, diagnostics.count(&diagnostic_store))
resolution := try resolve(mem.c_allocator, &tree, &diagnostic_store)
defer deinit(&resolution)
function :: ast.package_decl(&tree, 0)
view :: ast.function_decl(&tree, function)
outer :: ast.package_decl(&tree, 1)
first_same :: ast.package_decl(&tree, 3)
second_same :: ast.package_decl(&tree, 4)
outer_param :: ast.function_param(&tree, function, 0)
p :: ast.function_param(&tree, function, 1)
duplicate_p :: ast.function_param(&tree, function, 2)
first_outer :: ast.block_statement(&tree, view.body, 0)
second_outer :: ast.block_statement(&tree, view.body, 1)
local_p :: ast.block_statement(&tree, view.body, 2)
local_int :: ast.block_statement(&tree, view.body, 3)
local_pkg :: ast.block_statement(&tree, view.body, 4)
rejected [8]ast.NodeId :: [
outer_param,
duplicate_p,
first_outer,
second_outer,
local_p,
local_int,
local_pkg,
second_same,
]
for rejected |id| {
invalid_binding :: binding(&resolution, id)
try testing.expect_equal(BindingKind.invalid, invalid_binding.kind)
if (diagnostics.get(&diagnostic_store, invalid_binding.data.diagnostic)) |diagnostic| {
try testing.expect_equal(diagnostics.Code.duplicate_name, diagnostic.code)
name :: if (id == outer_param or id == duplicate_p)
ast.param(&tree, id).name
else
ast.statement_decl(&tree, id).name
try testing.expect_equal(name.start, diagnostic.span.start)
try testing.expect_equal(name.end, diagnostic.span.end)
} else {
return .expectation_failed
}
}
try testing.expect(
binding(&resolution, first_outer).data.diagnostic != binding(&resolution, second_outer).data.diagnostic,
)
try testing.expect(
binding(&resolution, outer_param).data.diagnostic != binding(&resolution, first_outer).data.diagnostic,
)
try testing.expect(
binding(&resolution, duplicate_p).data.diagnostic != binding(&resolution, local_p).data.diagnostic,
)
outer_ref :: ast.statement_expr(&tree, ast.block_statement(&tree, view.body, 5))
try testing.expect_equal(BindingKind.invalid, binding(&resolution, outer_ref).kind)
try testing.expect_equal(
binding(&resolution, outer_param).data.diagnostic,
binding(&resolution, outer_ref).data.diagnostic,
)
p_ref :: ast.statement_expr(&tree, ast.block_statement(&tree, view.body, 6))
try testing.expect_equal(BindingKind.declaration, binding(&resolution, p_ref).kind)
try testing.expect_equal(p, binding(&resolution, p_ref).data.declaration)
int_ref :: ast.statement_expr(&tree, ast.block_statement(&tree, view.body, 7))
try testing.expect_equal(BindingKind.invalid, binding(&resolution, int_ref).kind)
try testing.expect_equal(
binding(&resolution, local_int).data.diagnostic,
binding(&resolution, int_ref).data.diagnostic,
)
pkg_ref :: ast.statement_expr(&tree, ast.block_statement(&tree, view.body, 8))
try testing.expect_equal(BindingKind.invalid, binding(&resolution, pkg_ref).kind)
try testing.expect_equal(
binding(&resolution, local_pkg).data.diagnostic,
binding(&resolution, pkg_ref).data.diagnostic,
)
try testing.expect_equal(BindingKind.declaration, binding(&resolution, view.return_type).kind)
try testing.expect_equal(outer, binding(&resolution, view.return_type).data.declaration)
parameter_type :: ast.param(&tree, outer_param).type
try testing.expect_equal(BindingKind.builtin, binding(&resolution, parameter_type).kind)
try testing.expect_equal(BuiltinType.int, binding(&resolution, parameter_type).data.builtin)
same_ref :: ast.statement_decl(&tree, second_same).initializer
try testing.expect_equal(BindingKind.declaration, binding(&resolution, same_ref).kind)
try testing.expect_equal(first_same, binding(&resolution, same_ref).data.declaration)
missing :: ast.statement_decl(&tree, first_outer).initializer
try testing.expect_equal(BindingKind.invalid, binding(&resolution, missing).kind)
if (diagnostics.get(&diagnostic_store, binding(&resolution, missing).data.diagnostic)) |diagnostic| {
try testing.expect_equal(diagnostics.Code.unknown_name, diagnostic.code)
try testing.expect_equal(ast.node_span(&tree, missing).start, diagnostic.span.start)
try testing.expect_equal(ast.node_span(&tree, missing).end, diagnostic.span.end)
} else {
return .expectation_failed
}
try testing.expect_equal(9, diagnostics.count(&diagnostic_store))
}
preserves_syntax_diagnostics_and_resolves_shared_unknown_types_once test {
strpool.strings = strpool.init(mem.c_allocator)
defer strpool.deinit(&strpool.strings)
diagnostic_store := diagnostics.init(mem.c_allocator)
defer diagnostics.deinit(&diagnostic_store)
source ::
`@ broken
`main proc(a, b Missing) void {
` empty :: ()
` bad :: absent + absent
` valid :: later
` later :: 1
` discarded(1 2)
`}
scan_state := lexer.init(mem.c_allocator)
defer lexer.deinit(&scan_state)
try lexer.scan(&scan_state, source, &diagnostic_store)
tree := try parser.parse(mem.c_allocator, scan_state.tokens.items, source, &diagnostic_store)
defer ast.deinit(&tree)
syntax_count :: diagnostics.count(&diagnostic_store)
invalid_decl :: ast.package_decl(&tree, 0)
try testing.expect_equal(ast.DeclKind.invalid, ast.decl_kind(&tree, invalid_decl))
function :: ast.package_decl(&tree, 1)
view :: ast.function_decl(&tree, function)
empty :: ast.statement_decl(&tree, ast.block_statement(&tree, view.body, 0)).initializer
try testing.expect_equal(ast.ExprKind.invalid, ast.expr_kind(&tree, empty))
invalid_statement :: ast.block_statement(&tree, view.body, 4)
try testing.expect_equal(ast.StmtKind.invalid, ast.stmt_kind(&tree, invalid_statement))
resolution := try resolve(mem.c_allocator, &tree, &diagnostic_store)
defer deinit(&resolution)
invalid_nodes [3]ast.NodeId :: [invalid_decl, empty, invalid_statement]
for invalid_nodes |id| {
invalid_view :: ast.invalid(&tree, id)
invalid_binding :: binding(&resolution, id)
try testing.expect_equal(BindingKind.invalid, invalid_binding.kind)
try testing.expect_equal(invalid_view.diagnostic, invalid_binding.data.diagnostic)
}
a :: ast.function_param(&tree, function, 0)
b :: ast.function_param(&tree, function, 1)
shared_type :: ast.param(&tree, a).type
try testing.expect_equal(
binding(&resolution, shared_type).data.diagnostic,
binding(&resolution, ast.param(&tree, b).type).data.diagnostic,
)
bad :: ast.binary_expr(
&tree,
ast.statement_decl(&tree, ast.block_statement(&tree, view.body, 1)).initializer,
)
unknown_nodes [3]ast.NodeId :: [shared_type, bad.left, bad.right]
for unknown_nodes |id| {
invalid_binding :: binding(&resolution, id)
try testing.expect_equal(BindingKind.invalid, invalid_binding.kind)
if (diagnostics.get(&diagnostic_store, invalid_binding.data.diagnostic)) |diagnostic| {
try testing.expect_equal(diagnostics.Code.unknown_name, diagnostic.code)
try testing.expect_equal(ast.node_span(&tree, id).start, diagnostic.span.start)
try testing.expect_equal(ast.node_span(&tree, id).end, diagnostic.span.end)
} else {
return .expectation_failed
}
}
try testing.expect(
binding(&resolution, bad.left).data.diagnostic != binding(&resolution, bad.right).data.diagnostic,
)
valid :: ast.block_statement(&tree, view.body, 2)
later :: ast.block_statement(&tree, view.body, 3)
valid_ref :: ast.statement_decl(&tree, valid).initializer
try testing.expect_equal(BindingKind.declaration, binding(&resolution, valid_ref).kind)
try testing.expect_equal(later, binding(&resolution, valid_ref).data.declaration)
try testing.expect_equal(syntax_count + 3, diagnostics.count(&diagnostic_store))
}
+11 -8
View File
@@ -6,17 +6,19 @@ InternError :: enum { out_of_space }
StringId :: alias u32
NO_STR :: maxval!(StringId)
no_str :: maxval!(StringId)
# cross-cutting concern, hence global singleton (owned by main.hon)
STRINGS StringPool := undefined
#! global pool owned by main.hon; initialize before scanning and release with deinit.
strings StringPool := undefined
#! retains its allocator and owns interned copies; release all storage with deinit.
StringPool :: struct {
ids hashmap.StringHashMap(StringId) # string → id
strings arraylist.ArrayList([]u8) # id → owned string
allocator mem.Allocator
}
#! retains `allocator` for interned copies and indexes; the caller owns the pool.
init proc(allocator mem.Allocator) StringPool {
return StringPool{
ids = hashmap.init(allocator),
@@ -25,19 +27,19 @@ init proc(allocator mem.Allocator) StringPool {
}
}
#! releases interned strings and indexes with the retained allocator.
deinit proc(pool @mut StringPool) void {
hashmap.deinit(&pool.ids)
for pool.strings.items |str| {
mem.free(pool.allocator, str)
}
for (pool.strings.items) |str| mem.free(pool.allocator, str)
arraylist.deinit(&pool.strings)
}
#! copies new strings into pool-owned storage; IDs and borrowed strings last until deinit.
intern proc(pool @mut StringPool, str []u8) StringId ! (mem.AllocError | InternError) {
if hashmap.get(&pool.ids, str) |id| return id
if (pool.strings.items.len >= usize(NO_STR)) return .out_of_space
id StringId :: StringId(pool.strings.items.len)
if (pool.strings.items.len >= usize(no_str)) return .out_of_space
id :: StringId(pool.strings.items.len)
owned_str []mut u8 :: try mem.alloc(u8, pool.allocator, str.len)
errdefer mem.free(pool.allocator, owned_str)
@@ -56,6 +58,7 @@ intern proc(pool @mut StringPool, str []u8) StringId ! (mem.AllocError | InternE
return id
}
#! returns pool-owned text borrowed until deinit; the caller must not free it.
get_str proc(pool @StringPool, id StringId) ?[]u8 {
if (usize(id) >= pool.strings.items.len) return null
return pool.strings.items[usize(id)]
+1 -1
View File
@@ -2,7 +2,7 @@ import "@std/mem"
import "@std/testing"
handles_interning test {
pool StringPool := init(mem.c_allocator)
pool := init(mem.c_allocator)
defer deinit(&pool)
input [5]mut u8 := ['h', 'e', 'l', 'l', 'o']
+15 -8
View File
@@ -1,5 +1,7 @@
import "@std/mem"
#! owns item storage and retains its allocator; release storage with deinit.
#! items is a borrowed view invalidated by growth or deinit.
ArrayList proc($T type) type {
return struct {
items []mut T
@@ -8,6 +10,7 @@ ArrayList proc($T type) type {
}
}
#! retain `allocator` for later growth and cleanup; this call does not allocate.
init proc($T type, allocator mem.Allocator) ArrayList(T) {
return ArrayList(T) {
items = mem.empty(T),
@@ -16,6 +19,7 @@ init proc($T type, allocator mem.Allocator) ArrayList(T) {
}
}
#! release owned storage using the retained allocator and reset the list.
deinit proc($T type, list @mut ArrayList(T)) void {
allocation :: list.items.ptr[..list.capacity]
mem.free(list.allocator, allocation)
@@ -23,10 +27,13 @@ deinit proc($T type, list @mut ArrayList(T)) void {
list.capacity = 0
}
reserve proc($T type, list @mut ArrayList(T), min_capacity usize) void ! mem.AllocError {
if min_capacity <= list.capacity {
return
}
#! grow owned storage with the retained allocator; growth invalidates borrowed item views.
reserve proc(
$T type,
list @mut ArrayList(T),
min_capacity usize,
) void ! mem.AllocError {
if (min_capacity <= list.capacity) return
new_capacity := 8
if list.capacity >= 8 {
@@ -41,9 +48,9 @@ reserve proc($T type, list @mut ArrayList(T), min_capacity usize) void ! mem.All
new_capacity = min_capacity
}
length usize :: list.items.len
allocation []mut T :: list.items.ptr[..list.capacity]
grown []mut T :: mem.realloc(list.allocator, allocation, new_capacity) catch |_| {
length :: list.items.len
allocation :: list.items.ptr[..list.capacity]
grown :: mem.realloc(list.allocator, allocation, new_capacity) catch |_| {
return .out_of_memory
}
list.items = grown.ptr[..length]
@@ -52,7 +59,7 @@ reserve proc($T type, list @mut ArrayList(T), min_capacity usize) void ! mem.All
}
append proc($T type, list @mut ArrayList(T), value T) void ! mem.AllocError {
length usize :: list.items.len
length :: list.items.len
if length == maxval!(usize) {
return .out_of_memory
}
+8 -8
View File
@@ -1,11 +1,11 @@
# Build configuration surface for `brolang build` (v0).
#
# A project's `build.bro` imports this module and declares a top-level constant
# named `config` of type `BuildConfig`. `brolang build [root]` type-checks
# build.bro, reads the config, and writes root/build/name.
#
# Declarative and literal-only: one executable per build. List fields take an
# address-of an array literal (`&["raylib"]`) and default to empty.
#! build configuration surface for `brolang build` (v0).
#!
#! a project's `build.bro` imports this module and declares a top-level constant
#! named `config` of type `BuildConfig`. `brolang build [root]` type-checks
#! build.bro, reads the config, and writes root/build/name.
#!
#! declarative and literal-only: one executable per build. list fields take an
#! address-of an array literal (`&["raylib"]`) and default to empty.
BuildConfig :: struct {
name []u8 # output executable name under root/build
source []u8 # program package directory, relative to build.bro
+5 -11
View File
@@ -17,23 +17,17 @@ print proc($format []u8, $Args type, args Args) void {
handle = io.Handle{ file_desc = c_int(io.Stream.stderr) },
write = write,
}
io.print(writer, format, Args, args) catch |_| {}
io.print(format, Args, writer, args) catch |_| {}
}
@hide
@hide:file
write proc(_ ?@mut anyopaque, handle io.Handle, bytes []u8) usize ! io.WriteError {
request := bytes.len
maximum :: usize(maxval!(c_long))
if request > maximum {
request = maximum
}
if (request > maximum) request = maximum
while true {
count :: c.write(handle.file_desc, bytes.ptr, c_ulong(request))
if count >= 0 {
return usize(count)
}
if c.__error()?^ != c.EINTR {
return .write_failed
}
if (count >= 0) return usize(count)
if (c.__error()?^ != c.EINTR) return .write_failed
}
}
+1 -1
View File
@@ -4,7 +4,7 @@ EnumMap proc($E, $V type) type {
match typeinfo!(E) {
.enum |info|: return struct {
present [info.fields.len]mut bool # fixme: replace with bitset
values [info.fields.len]mut V
values [info.fields.len]mut V
}
else: compile_error!("EnumMap key must be an enum")
}
+2 -1
View File
@@ -1,6 +1,7 @@
import "@std/mem"
import "@std/testing"
@hide:file
TestEnum :: enum(u8) {
ident = 3
int = 8
@@ -8,7 +9,7 @@ TestEnum :: enum(u8) {
}
handles_sparse_enum_get test {
names EnumMap(TestEnum, []u8) := init({
names EnumMap(TestEnum, []u8) :: init({
ident = "identifier",
int = "integer",
})
+8 -10
View File
@@ -11,6 +11,8 @@ Entry proc($K, $V type) type {
}
}
#! owns entry storage and retains its allocator; release storage with deinit.
#! keys and values are stored by value; any resources they reference remain caller-owned.
HashMap proc(
$K, $V type,
$hash_key proc(key K) usize,
@@ -27,6 +29,7 @@ StringHashMap proc($V type) type {
return HashMap([]u8, V, str_hash, str_eql)
}
#! retain `allocator` for later growth and cleanup; this call does not allocate.
init proc(
$K, $V type,
$hash_key proc(key K) usize,
@@ -40,7 +43,7 @@ init proc(
}
}
#! free the entries in the hash map.
#! free owned entry storage using the retained allocator, not resources referenced by keys or values.
#! note: this operation invalidates the map.
deinit proc(
$K, $V type,
@@ -81,21 +84,17 @@ put proc(
key K,
value V,
) void ! (PutError | mem.AllocError) {
# check grow
threshold :: map.entries.len - divtrunc!(map.entries.len, 4)
if (map.entries.len == 0 or map.count + 1 > threshold) {
old_entries :: map.entries
new_size :: if (old_entries.len > 0) old_entries.len * 2 else 8
new_entries :: try mem.alloc(Entry(K, V), map.allocator, new_size)
# zero new entries
for (0..new_entries.len) |i| new_entries[i].hash = 0
# move old entries
for old_entries |entry| {
if (entry.hash == 0) continue
# find an empty slot
idx := entry.hash & (new_entries.len - 1)
while new_entries[idx].hash != 0 {
idx = (idx + 1) & (new_entries.len - 1)
@@ -108,7 +107,6 @@ put proc(
mem.free(map.allocator, old_entries)
}
# put new entry
hash :: normalize(hash_key(key))
idx := hash & (map.entries.len - 1)
@@ -132,7 +130,7 @@ put proc(
}
}
@hide
@hide:file
normalize proc(hash usize) usize {
# mapping both 0 and 1 to 1 is safe because equality resolves
# collisions (since hash and key must both be equal).
@@ -142,10 +140,10 @@ normalize proc(hash usize) usize {
#! FNV-1a hash implementation.
#! note: vulnerable to collision attacks.
@hide
@hide:file
str_hash proc(key []u8) usize {
hash u32 := 2166136261 # offset basis
prime u32 := 16777619
prime u32 :: 16777619
for key |byte| {
product u64 :: u64(hash xor u32(byte)) * prime
@@ -155,5 +153,5 @@ str_hash proc(key []u8) usize {
return usize(hash)
}
@hide
@hide:file
str_eql proc(a, b []u8) bool { return mem.eql(a, b) }
+20 -8
View File
@@ -19,15 +19,18 @@ CloseError :: enum {
close_failed
}
#! open a caller-owned handle; release it with close.
open proc(io Io, path [;0]u8, mode FileMode) File ! OpenError {
handle Handle :: try io.vtable.open(io.context, path, mode)
return File{ io = io, handle = handle }
}
#! close the owned handle, invalidating its borrowed readers and writers.
close proc(file File) void ! CloseError {
try file.io.vtable.close(file.io.context, file.handle)
}
#! borrow the file handle; the reader must not outlive the open file.
reader proc(file File) Reader {
return Reader {
context = file.io.context,
@@ -36,6 +39,7 @@ reader proc(file File) Reader {
}
}
#! borrow the file handle; the writer must not outlive the open file.
writer proc(file File) Writer {
return Writer {
context = file.io.context,
@@ -44,7 +48,8 @@ writer proc(file File) Writer {
}
}
@hide system_read proc(_ ?@mut anyopaque, handle Handle, buffer []mut u8) usize ! ReadError {
@hide:file
system_read proc(_ ?@mut anyopaque, handle Handle, buffer []mut u8) usize ! ReadError {
request := buffer.len
maximum usize :: usize(maxval!(c_long))
if (request > maximum) request = maximum
@@ -60,7 +65,8 @@ writer proc(file File) Writer {
}
}
@hide system_write proc(_ ?@mut anyopaque, handle Handle, bytes []u8) usize ! WriteError {
@hide:file
system_write proc(_ ?@mut anyopaque, handle Handle, bytes []u8) usize ! WriteError {
request := bytes.len
maximum usize :: usize(maxval!(c_long))
if (request > maximum) request = maximum
@@ -76,7 +82,8 @@ writer proc(file File) Writer {
}
}
@hide system_open proc(_ ?@mut anyopaque, path [;0]u8, mode FileMode) Handle ! OpenError {
@hide:file
system_open proc(_ ?@mut anyopaque, path [;0]u8, mode FileMode) Handle ! OpenError {
flags := c.O_RDONLY
match mode {
.read_only: flags = c.O_RDONLY
@@ -90,23 +97,28 @@ writer proc(file File) Writer {
}
}
@hide system_close proc(_ ?@mut anyopaque, handle Handle) void ! CloseError {
@hide:file
system_close proc(_ ?@mut anyopaque, handle Handle) void ! CloseError {
if (c.close(handle.file_desc) != 0) return .close_failed
}
@hide system_stdin proc(_ ?@mut anyopaque) Handle {
@hide:file
system_stdin proc(_ ?@mut anyopaque) Handle {
return Handle{ file_desc = c_int(Stream.stdin) }
}
@hide system_stdout proc(_ ?@mut anyopaque) Handle {
@hide:file
system_stdout proc(_ ?@mut anyopaque) Handle {
return Handle{ file_desc = c_int(Stream.stdout) }
}
@hide system_stderr proc(_ ?@mut anyopaque) Handle {
@hide:file
system_stderr proc(_ ?@mut anyopaque) Handle {
return Handle{ file_desc = c_int(Stream.stderr) }
}
@hide system_vtable IoVTable :: IoVTable {
@hide:file
system_vtable IoVTable :: IoVTable {
read = system_read,
write = system_write,
open = system_open,
+33 -14
View File
@@ -1,4 +1,6 @@
import "@ffi/c"
import "@std/mem"
# typeinfo! requires this import even without a qualified meta reference.
import "@std/meta"
ReadError :: enum {
@@ -23,6 +25,7 @@ Stream :: enum(c_int) {
stderr = c.STDERR_FILENO
}
#! context and vtable are borrowed and must outlive derived readers and writers.
Io :: struct {
context ?@mut anyopaque
vtable @IoVTable
@@ -38,12 +41,14 @@ IoVTable :: struct {
stderr @proc(context ?@mut anyopaque) Handle
}
#! borrows its context and handle; it does not close the handle.
Reader :: struct {
context ?@mut anyopaque
handle Handle
read @proc(context ?@mut anyopaque, handle Handle, buffer []mut u8) usize ! ReadError
}
#! borrows its context and handle; it does not close the handle.
Writer :: struct {
context ?@mut anyopaque
handle Handle
@@ -101,7 +106,7 @@ stderr proc(io Io) Writer {
}
}
print proc(output Writer, $format []u8, $Args type, args Args) void ! WriteError {
print proc($format []u8, $Args type, output Writer, args Args) void ! WriteError {
inline for parse_format(format.len, format, Args) |token| {
match token.kind {
.unused: break
@@ -120,14 +125,19 @@ print proc(output Writer, $format []u8, $Args type, args Args) void ! WriteError
}
@hide:file
write_integer_signed proc(output Writer, value i64, base u64, uppercase bool) void ! WriteError {
write_integer_signed proc(
output Writer,
value i64,
base u64,
uppercase bool,
) void ! WriteError {
buffer [65]mut u8 := undefined
end := buffer.len
current := value
while true {
digit_value i64 :: rem!(current, i64(base))
digit := if (digit_value < 0)
digit :: if (digit_value < 0)
u8(-digit_value)
else
u8(digit_value)
@@ -152,7 +162,12 @@ write_integer_signed proc(output Writer, value i64, base u64, uppercase bool) vo
}
@hide:file
write_integer_unsigned proc(output Writer, value u64, base u64, uppercase bool) void ! WriteError {
write_integer_unsigned proc(
output Writer,
value u64,
base u64,
uppercase bool,
) void ! WriteError {
buffer [65]mut u8 := undefined
end := buffer.len
current := value
@@ -201,9 +216,7 @@ FormatToken :: struct {
@hide:file
parse_format proc($N usize, $format []u8, $Args type) [N]mut FormatToken {
tokens [N]mut FormatToken := undefined
for (usize(0))..format.len |index| {
tokens[index] = FormatToken{ kind = .unused, start = 0, end = 0, field = "" }
}
mem.set(FormatToken, tokens[..], FormatToken{ kind = .unused, start = 0, end = 0, field = "" })
field_count := 0
match typeinfo!(Args) {
@@ -314,7 +327,7 @@ parse_format proc($N usize, $format []u8, $Args type) [N]mut FormatToken {
if literal_start < format.len {
tokens[token_count] = FormatToken{
kind = .literal,
start = literal_start,
start = literal_start,
end = format.len,
field = "",
}
@@ -351,7 +364,13 @@ scalar_or_distinct_type proc($T type) bool {
}
@hide:file
write_integer proc(output Writer, $T type, value T, base u64, uppercase bool) void ! WriteError {
write_integer proc(
$T type,
output Writer,
value T,
base u64,
uppercase bool,
) void ! WriteError {
match typeinfo!(T) {
.integer: if minval!(T) < 0 {
try write_integer_signed(output, i64(value), base, uppercase)
@@ -369,7 +388,7 @@ write_integer proc(output Writer, $T type, value T, base u64, uppercase bool) vo
# note: libc keeps float formatting small; replace it with a native shortest-roundtrip writer if locale independence matters.
@hide:file
write_float proc(output Writer, $T type, value T, scientific bool) void ! WriteError {
write_float proc($T type, output Writer, value T, scientific bool) void ! WriteError {
match typeinfo!(T) {
.float: {
buffer [64]mut u8 := undefined
@@ -395,7 +414,7 @@ write_float proc(output Writer, $T type, value T, scientific bool) void ! WriteE
}
@hide:file
write_decimal proc(output Writer, $T type, value T) void ! WriteError {
write_decimal proc($T type, output Writer, value T) void ! WriteError {
match typeinfo!(T) {
.integer: try write_integer(output, value, 10, false)
.float: try write_float(output, value, false)
@@ -409,13 +428,13 @@ write_decimal proc(output Writer, $T type, value T) void ! WriteError {
}
@hide:file
write_character proc(output Writer, $T type, value T) void ! WriteError {
write_character proc($T type, output Writer, value T) void ! WriteError {
match typeinfo!(T) {
.integer: {
if minval!(T) < 0 or maxval!(T) > 255 {
compile_error!("io.print '{c}' requires an unsigned integer that fits in u8")
}
buffer [1]u8 := [u8(value)]
buffer [1]u8 :: [u8(value)]
try write_all(output, buffer[..])
}
.distinct |backing|: if scalar_or_distinct_type(backing) {
@@ -428,7 +447,7 @@ write_character proc(output Writer, $T type, value T) void ! WriteError {
}
@hide:file
write_default proc(output Writer, $T type, value T) void ! WriteError {
write_default proc($T type, output Writer, value T) void ! WriteError {
match typeinfo!(T) {
.bool: if value {
try write_all(output, "true")
+53 -21
View File
@@ -4,6 +4,7 @@ AllocError :: enum {
out_of_memory
}
#! allocator context and vtable are borrowed and must outlive allocations made through them.
Allocator :: struct {
context ?@mut anyopaque
vtable @AllocatorVTable
@@ -15,38 +16,62 @@ AllocatorVTable :: struct {
free @proc(context ?@mut anyopaque, memory ?*mut u8, size usize, alignment usize) void
}
raw_alloc proc(allocator Allocator, size usize, alignment usize) ?*mut u8 {
#! returned storage belongs to the caller; release it with raw_free using the same allocator, size, and alignment.
raw_alloc proc(
allocator Allocator,
size usize,
alignment usize,
) ?*mut u8 {
return allocator.vtable.alloc(allocator.context, size, alignment)
}
raw_realloc proc(allocator Allocator, memory ?*mut u8, old_size usize, new_size usize, alignment usize) ?*mut u8 {
#! resize caller-owned storage with its original allocator and alignment.
#! on allocation failure with nonzero `new_size`, original storage remains caller-owned; release the result with raw_free.
raw_realloc proc(
allocator Allocator,
memory ?*mut u8,
old_size usize,
new_size usize,
alignment usize,
) ?*mut u8 {
return allocator.vtable.realloc(allocator.context, memory, old_size, new_size, alignment)
}
raw_free proc(allocator Allocator, memory ?*mut u8, size usize, alignment usize) void {
#! release caller-owned storage with its original allocator, size, and alignment.
raw_free proc(
allocator Allocator,
memory ?*mut u8,
size usize,
alignment usize,
) void {
allocator.vtable.free(allocator.context, memory, size, alignment)
}
eql proc($T type, left, right []T) bool {
if (left.len != right.len) return false
for (0..left.len) |i| if (left[i] != right[i]) {
return false
}
for (0..left.len) |i| if (left[i] != right[i]) return false
return true
}
#! assign `value` to each slice element, preserving its typed value rather than repeating a byte.
#! an empty `destination` is a no-op.
set proc($T type, destination []mut T, value T) void {
for (0..destination.len) |i| destination[i] = value
}
#! allocate memory for a slice of type `T` with `count` elements.
#! returned storage belongs to the caller; release it with free using the same allocator.
alloc proc($T type, allocator Allocator, count usize) []mut T ! AllocError {
if (count == 0) return empty_slice(T, 0)
element_size usize :: sizeof!(T)
element_size :: sizeof!(T)
if (element_size == 0) return empty_slice(T, count)
if count > divtrunc!(maxval!(usize), element_size) {
return .out_of_memory
}
memory := raw_alloc(allocator, count * element_size, alignof!(T))
memory :: raw_alloc(allocator, count * element_size, alignof!(T))
if memory |bytes| {
pointer *mut T :: ptrcast!(bytes)
return pointer[..count]
@@ -58,17 +83,22 @@ alloc proc($T type, allocator Allocator, count usize) []mut T ! AllocError {
#! reallocate memory for a slice of type `T` with `new_count` elements.
#! reallocating with `new_count == 0` will free the memory and return an empty slice.
#! note: memory must be reallocated with the same allocator that was used to allocate it.
realloc proc($T type, allocator Allocator, memory []mut T, new_count usize) []mut T ! AllocError {
if new_count == memory.len {
return memory
}
#! the result belongs to the caller; release it with free using that allocator.
#! on failure the original slice remains valid and caller-owned.
realloc proc(
$T type,
allocator Allocator,
memory []mut T,
new_count usize,
) []mut T ! AllocError {
if (new_count == memory.len) return memory
if new_count == 0 {
free(allocator, memory)
return empty_slice(T, 0)
}
element_size usize :: sizeof!(T)
element_size :: sizeof!(T)
if (element_size == 0) return empty_slice(T, new_count)
if (new_count > divtrunc!(maxval!(usize), element_size)) return .out_of_memory
@@ -79,7 +109,7 @@ realloc proc($T type, allocator Allocator, memory []mut T, new_count usize) []mu
old_size = memory.len * element_size
}
resized := raw_realloc(
resized :: raw_realloc(
allocator,
old_memory,
old_size,
@@ -99,8 +129,9 @@ realloc proc($T type, allocator Allocator, memory []mut T, new_count usize) []mu
#! note: memory must be freed with the same allocator that was used to allocate it.
free proc($T type, allocator Allocator, memory []T) void {
if (memory.len == 0 or sizeof!(T) == 0) return
raw_free(allocator, ptrcast!(
constcast!(memory).ptr),
raw_free(
allocator,
ptrcast!(constcast!(memory).ptr),
memory.len * sizeof!(T),
alignof!(T),
)
@@ -116,14 +147,15 @@ empty_slice proc($T type, count usize) []mut T {
}
#! get an empty slice of type `T` with 0 elements.
#! the view borrows shared empty storage and requires no cleanup.
empty proc($T type) []mut T {
return empty_slice(T, 0)
}
@hide
@hide:file
empty_storage [1]mut u64 := [0]
@hide
@hide:file
malloc_alignment usize :: 16 # note: aarch64-macos libc malloc alignment assumption.
@hide:file
@@ -131,7 +163,7 @@ power_of_two proc(value usize) bool {
if (value == 0) return false
current := value
while current > 1 {
half := divtrunc!(current, 2)
half :: divtrunc!(current, 2)
if (half * 2 != current) return false
current = half
}
@@ -144,7 +176,7 @@ c_alloc proc(_ ?@mut anyopaque, size usize, alignment usize) ?*mut u8 {
if (alignment <= malloc_alignment) return ptrcast!(c.malloc(c_ulong(size)))
memory [1]mut ?*mut anyopaque := [null]
status := c.posix_memalign((&memory).ptr, c_ulong(alignment), c_ulong(size))
status :: c.posix_memalign((&memory).ptr, c_ulong(alignment), c_ulong(size))
if (status != 0) return null
return ptrcast!(memory[0])
@@ -170,7 +202,7 @@ c_realloc proc(
return ptrcast!(c.realloc(old_memory, c_ulong(new_size)))
}
new_memory := c_alloc(null, new_size, alignment)
new_memory :: c_alloc(null, new_size, alignment)
if new_memory |new_bytes| {
copy_size := old_size
if (new_size < copy_size) copy_size = new_size
+27
View File
@@ -0,0 +1,27 @@
import "@std/testing"
@hide:file
Record :: struct {
tag u8
id u32
}
fills_typed_values_within_slice_boundaries test {
values [4]mut u32 := [1, 2, 3, 4]
set(values[1..3], u32(305419896))
set(values[2..2], u32(0))
try testing.expect_equal(u32(1), values[0])
try testing.expect_equal(u32(305419896), values[1])
try testing.expect_equal(u32(305419896), values[2])
try testing.expect_equal(u32(4), values[3])
records [2]mut Record := [
Record{ tag = 0, id = 0 },
Record{ tag = 0, id = 0 },
]
set(records[..], Record{ tag = 3, id = maxval!(u32) })
for records |record| {
try testing.expect_equal(u8(3), record.tag)
try testing.expect_equal(maxval!(u32), record.id)
}
}
+10 -4
View File
@@ -1,18 +1,24 @@
import "@std/testing"
@hide:file
TestTokenKind :: enum(u8) {
ident = 3
int = 8
eof = 21
}
@hide:file
TestNames :: alias EnumFieldStruct(TestTokenKind, ?[]u8, some!(null))
@hide:file
TestArrayAlias :: alias [3]u16
@hide:file
TestInner :: distinct u16
@hide:file
TestOuter :: distinct TestInner
@hide:file
TestOuterAlias :: alias TestOuter
@hide
@hide:file
array_info_matches proc($Array, $Child type, $len usize) bool {
return match typeinfo!(Array) {
.array |info|: info.child == Child and info.len == len
@@ -20,7 +26,7 @@ array_info_matches proc($Array, $Child type, $len usize) bool {
}
}
@hide
@hide:file
distinct_info_matches proc($Distinct, $Backing type) bool {
return match typeinfo!(Distinct) {
.distinct |backing|: backing == Backing
@@ -43,7 +49,7 @@ distinct_reflection_exposes_immediate_backing test {
}
enum_field_struct_defaults test {
names TestNames := {
names TestNames :: {
ident = "identifier",
int = "integer",
}
@@ -60,6 +66,6 @@ enum_field_struct_defaults test {
if (field!(names, "eof")) |_| try testing.expect(false)
empty TestNames := {}
empty TestNames :: {}
if (field!(empty, "ident")) |_| try testing.expect(false)
}
+3 -2
View File
@@ -16,6 +16,7 @@ Pair proc($V type) type {
}
#! initializes a static string map from a list of key-value pairs (constructed at compile-time).
#! returned views use compile-time storage and require no cleanup.
init proc($V type, $N usize, $entries [N]Pair(V)) StaticStringMap(V) {
if N > usize(maxval!(u32)) {
compile_error!("static string map has too many entries")
@@ -39,7 +40,7 @@ init proc($V type, $N usize, $entries [N]Pair(V)) StaticStringMap(V) {
}
if N == 0 {
len_indexes [0]u32 := undefined
len_indexes [0]u32 :: undefined
return StaticStringMap(V){
keys = keys[..],
values = values[..],
@@ -83,7 +84,7 @@ init proc($V type, $N usize, $entries [N]Pair(V)) StaticStringMap(V) {
get proc($V type, map @StaticStringMap(V), key []u8) ?V {
if (map.keys.len == 0 or key.len > maxval!(u32)) return null
length := u32(key.len)
length :: u32(key.len)
if (length < map.min_len or length > map.max_len) return null
idx := usize(map.len_indexes[usize(length)])