package parser import "../ast" import "../source" import "../symbol" import "../token" import "../types" import "base:intrinsics" import "core:fmt" import "core:strconv" import "core:strings" import "core:unicode/utf8" Parser :: struct { tokens: ^token.Stream, source_file: ^source.Source, diagnostics: ^source.Diagnostics, module: ast.Module, pkg: ast.Package_Id, file: ast.File_Id, cursor: int, delimiter_depth: int, range_disabled: int, // Suppresses `Name { ... }` struct-literal parsing at delimiter depth 0 so a // control-flow condition like `if foo { ... }` does not swallow the block as a // struct literal. Nested `(`/`[`/call-arg contexts (delimiter_depth > 0) still // allow struct literals. no_struct_literal: bool, hidden_names: [dynamic]symbol.Id, } MAX_EXPRESSION_NESTING :: 256 token_text :: proc(parser: ^Parser, tok: token.Token) -> string { if tok.span.end < tok.span.start || int(tok.span.end) > len(parser.source_file.text) { return "" } return parser.source_file.text[int(tok.span.start):int(tok.span.end)] } file_hidden_name :: proc(parser: ^Parser, name: symbol.Id) -> bool { for hidden in parser.hidden_names { if hidden == name { return true } } return false } collect_hidden_names :: proc(parser: ^Parser) { depth := 0 for item, index in parser.tokens.items { #partial switch item.kind { case .Left_Brace: depth += 1 case .Right_Brace: depth = max(depth-1, 0) case .Keyword_Hide: if depth == 0 && index+1 < len(parser.tokens.items) && parser.tokens.items[index+1].kind == .Identifier { append(&parser.hidden_names, parser.tokens.items[index+1].symbol) } case: } } } span_from :: proc(first, last: source.Span) -> source.Span { return source.Span{file=first.file, start=first.start, end=last.end} } current :: proc(parser: ^Parser) -> token.Token { return parser.tokens.items[min(parser.cursor, len(parser.tokens.items)-1)] } previous :: proc(parser: ^Parser) -> token.Token { return parser.tokens.items[max(parser.cursor-1, 0)] } peek :: proc(parser: ^Parser) -> token.Token { return parser.tokens.items[min(parser.cursor+1, len(parser.tokens.items)-1)] } // A loop body may be labeled `blk: { ... }` so a nested `yield :blk x` can exit // it past an enclosing `if`. Consumes and returns the label when the next tokens // are `Identifier Colon`; otherwise leaves the cursor untouched. parse_optional_loop_label :: proc(parser: ^Parser) -> symbol.Id { if current(parser).kind == .Identifier && peek(parser).kind == .Colon { name := advance(parser) // the label name advance(parser) // consume ':' skip_newlines(parser) return name.symbol } return symbol.INVALID } advance :: proc(parser: ^Parser) -> token.Token { result := current(parser) if result.kind != .Eof { parser.cursor += 1 } return result } allow :: proc(parser: ^Parser, kind: token.Kind) -> (token.Token, bool) { if current(parser).kind == kind { return advance(parser), true } return current(parser), false } parse_member_name :: proc(parser: ^Parser, allow_keyword := true) -> (token.Token, bool) { name := current(parser) if name.kind != .Identifier && (!allow_keyword || !token.is_keyword(name.kind)) { return name, false } advance(parser) if name.kind != .Identifier { name.symbol = symbol.intern(parser.tokens.symbols, token_text(parser, name)) } return name, true } skip_newlines :: proc(parser: ^Parser) { for current(parser).kind == .Newline { advance(parser) } } add_expr :: proc(parser: ^Parser, expr: ast.Expr) -> ast.Expr_Id { id := ast.expr_id(len(parser.module.exprs)) append(&parser.module.exprs, expr) return id } invalid_expr :: proc(parser: ^Parser, span: source.Span, message: string) -> ast.Expr_Id { id := source.add(parser.diagnostics, span, message) return add_expr(parser, ast.Expr{ kind=.Invalid, span=span, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=id, }) } is_type_token :: proc(kind: token.Kind) -> bool { #partial switch kind { case .Keyword_Int, .Keyword_Uint, .Keyword_Float, .Keyword_Range, .Keyword_I8, .Keyword_I16, .Keyword_I32, .Keyword_I64, .Keyword_U8, .Keyword_U16, .Keyword_U32, .Keyword_U64, .Keyword_Isize, .Keyword_Usize, .Keyword_F32, .Keyword_F64, .Keyword_C_Char, .Keyword_C_Schar, .Keyword_C_Uchar, .Keyword_C_Short, .Keyword_C_Ushort, .Keyword_C_Int, .Keyword_C_Uint, .Keyword_C_Long, .Keyword_C_Ulong, .Keyword_C_Longlong, .Keyword_C_Ulonglong, .Keyword_C_Float, .Keyword_C_Double, .Keyword_C_Longdouble, .Keyword_Void, .Keyword_Anyopaque, .Keyword_Bool, .Keyword_Func, .Keyword_C_Func, .Identifier, .Question, .At, .Star, .Left_Bracket: return true } return false } decode_character :: proc(parser: ^Parser, tok: token.Token) -> (u64, bool) { text := token_text(parser, tok) if len(text) < 3 { return 0, false } contents := text[1:len(text)-1] if len(contents) == 2 && contents[0] == '\\' { switch contents[1] { case '0': return 0, true case 'n': return '\n', true case 'r': return '\r', true case 't': return '\t', true case '\\': return '\\', true case '\'': return '\'', true } return 0, false } value, width := utf8.decode_rune_in_string(contents) return u64(value), width == len(contents) } parse_type_constant :: proc(parser: ^Parser) -> (u64, bool) { negative := false if _, ok := allow(parser, .Minus); ok { negative = true } tok := current(parser) if tok.kind == .Integer { advance(parser) value, ok := parse_integer_magnitude(token_text(parser, tok)) if !ok { return 0, false } if negative { return transmute(u64)-i64(value), true } return value, true } if !negative && tok.kind == .Character { advance(parser) return decode_character(parser, tok) } source.add(parser.diagnostics, tok.span, "expected an integer or character constant") return 0, false } parse_type_atom :: proc(parser: ^Parser) -> ast.Type_Syntax { tok := current(parser) if tok.kind == .Question { advance(parser) child := parse_type(parser) return types.intern(&parser.module.type_store, types.Node{kind=.Optional, child=child}) } if tok.kind == .At || tok.kind == .Star { many := tok.kind == .Star advance(parser) _, mutable := allow(parser, .Keyword_Mut) child := parse_type(parser) return types.intern(&parser.module.type_store, types.Node{ kind=.Pointer, child=child, mutable=mutable, many=many, }) } if tok.kind == .Left_Bracket { advance(parser) node := types.Node{} if _, ok := allow(parser, .Right_Bracket); ok { node.kind = .Slice } else if _, ok := allow(parser, .Star); ok { node.kind = .Pointer node.many = true node.has_sentinel = true if _, ok = allow(parser, .Semicolon); !ok { source.add(parser.diagnostics, current(parser).span, "expected ';' after '*' in sentinel pointer type") } node.sentinel, _ = parse_type_constant(parser) if _, ok = allow(parser, .Right_Bracket); !ok { source.add(parser.diagnostics, current(parser).span, "expected ']' after sentinel pointer type") } } else if _, ok := allow(parser, .Semicolon); ok { node.kind = .Slice node.has_sentinel = true node.sentinel, _ = parse_type_constant(parser) if _, ok = allow(parser, .Right_Bracket); !ok { source.add(parser.diagnostics, current(parser).span, "expected ']' after sentinel slice type") } } else { node.kind = .Array if _, ok := allow(parser, .Underscore); ok { node.inferred_count = true } else { if (current(parser).kind == .Integer || current(parser).kind == .Character) && (peek(parser).kind == .Right_Bracket || peek(parser).kind == .Semicolon) { count, ok := parse_type_constant(parser) node.count = count _ = ok } else { expr := parse_expression(parser) node.count_expr = u32(expr) node.unresolved_count = true } } if _, ok := allow(parser, .Semicolon); ok { node.has_sentinel = true node.sentinel, _ = parse_type_constant(parser) } if _, ok := allow(parser, .Right_Bracket); !ok { source.add(parser.diagnostics, current(parser).span, "expected ']' after array type") } } _, node.mutable = allow(parser, .Keyword_Mut) node.child = parse_type(parser) return types.intern(&parser.module.type_store, node) } #partial switch tok.kind { case .Keyword_Int: advance(parser) return types.INT case .Keyword_Uint: advance(parser) return types.UINT case .Keyword_Float: advance(parser) return types.FLOAT case .Keyword_Range: advance(parser) return types.RANGE case .Keyword_I8: advance(parser) return types.I8 case .Keyword_I16: advance(parser) return types.I16 case .Keyword_I32: advance(parser) return types.I32 case .Keyword_I64: advance(parser) return types.I64 case .Keyword_U8: advance(parser) return types.U8 case .Keyword_U16: advance(parser) return types.U16 case .Keyword_U32: advance(parser) return types.U32 case .Keyword_U64: advance(parser) return types.U64 case .Keyword_Isize: advance(parser) return types.ISIZE case .Keyword_Usize: advance(parser) return types.USIZE case .Keyword_F32: advance(parser) return types.F32 case .Keyword_F64: advance(parser) return types.F64 case .Keyword_C_Char: advance(parser) return types.C_CHAR case .Keyword_C_Schar: advance(parser) return types.C_SCHAR case .Keyword_C_Uchar: advance(parser) return types.C_UCHAR case .Keyword_C_Short: advance(parser) return types.C_SHORT case .Keyword_C_Ushort: advance(parser) return types.C_USHORT case .Keyword_C_Int: advance(parser) return types.C_INT case .Keyword_C_Uint: advance(parser) return types.C_UINT case .Keyword_C_Long: advance(parser) return types.C_LONG case .Keyword_C_Ulong: advance(parser) return types.C_ULONG case .Keyword_C_Longlong: advance(parser) return types.C_LONGLONG case .Keyword_C_Ulonglong: advance(parser) return types.C_ULONGLONG case .Keyword_C_Float: advance(parser) return types.C_FLOAT case .Keyword_C_Double: advance(parser) return types.C_DOUBLE case .Keyword_C_Longdouble: advance(parser) return types.C_LONGDOUBLE case .Keyword_Void: advance(parser) return types.VOID case .Keyword_Anyopaque: advance(parser) return types.ANYOPAQUE case .Keyword_Bool: advance(parser) return types.BOOL case .Keyword_Func, .Keyword_C_Func: c_abi := tok.kind == .Keyword_C_Func advance(parser) if _, ok := allow(parser, .Left_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected '(' after function type") return types.INVALID } params, variadic := parse_params(parser) if _, ok := allow(parser, .Right_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after function type parameters") } result := parse_type(parser) if _, ok := allow(parser, .Bang); ok { error_type := parse_error_type(parser) if c_abi { source.add(parser.diagnostics, current(parser).span, "c_func pointer types cannot be fallible") } else { result = types.fallible(&parser.module.type_store, result, error_type) } } param_types := make([]types.Type, len(params), parser.module.allocator) for param, index in params { param_types[index] = param.type } function_type := types.function(&parser.module.type_store, param_types, result, c_abi, variadic) delete(param_types, parser.module.allocator) delete(params, parser.module.allocator) return function_type case .Identifier: first := advance(parser) name := first qualifier := symbol.INVALID if _, ok := allow(parser, .Dot); ok { qualifier = first.symbol if current(parser).kind != .Identifier { source.add(parser.diagnostics, current(parser).span, "expected a type name after '.'") return types.INVALID } name = advance(parser) } named := types.named( &parser.module.type_store, u32(parser.pkg), u32(name.symbol), u32(qualifier), u32(parser.file), !symbol.is_valid(qualifier) && file_hidden_name(parser, name.symbol), ) if current(parser).kind == .Left_Paren { call := parse_call(parser, qualifier, first, name, 0, false) return types.intern(&parser.module.type_store, types.Node{ kind=.Type_Call, count_expr=u32(call), }) } append(&parser.module.type_uses, ast.Type_Use{ type=named, span=first.span, pkg=parser.pkg, file=parser.file, diagnostic=source.INVALID_DIAGNOSTIC, }) return named } source.add(parser.diagnostics, tok.span, "expected a type") return types.INVALID } parse_type_pipe_tail :: proc(parser: ^Parser, left: ast.Type_Syntax) -> ast.Type_Syntax { result := left for current(parser).kind == .Pipe { operator := advance(parser) right := parse_type_atom(parser) composed, compose_error := types.compose_sum(&parser.module.type_store, result, right) if compose_error == .Unsupported { source.add(parser.diagnostics, operator.span, "only native unbacked enums and tagged unions can be composed with '|'") result = types.INVALID } else if compose_error == .Conflict { source.add(parser.diagnostics, operator.span, "sum composition contains the same variant name with different payload types") result = types.INVALID } else { result = composed } } return result } parse_type :: proc(parser: ^Parser) -> ast.Type_Syntax { return parse_type_pipe_tail(parser, parse_type_atom(parser)) } parse_error_type :: proc(parser: ^Parser) -> ast.Type_Syntax { left := types.INVALID #partial switch current(parser).kind { case .Keyword_Enum: left = parse_inline_enum_type(parser) case .Keyword_Union: left = parse_inline_union_type(parser) case: left = parse_type_atom(parser) } return parse_type_pipe_tail(parser, left) } skip_parenthesized :: proc(parser: ^Parser) -> source.Span { start := current(parser) depth := 0 end := start for current(parser).kind != .Eof { tok := advance(parser) end = tok if tok.kind == .Left_Paren { depth += 1 } else if tok.kind == .Right_Paren { depth -= 1 if depth == 0 { break } } } return span_from(start.span, end.span) } parse_call_args :: proc(parser: ^Parser, nesting: int) -> ([]ast.Expr_Id, token.Token) { left_paren := advance(parser) parser.delimiter_depth += 1 defer parser.delimiter_depth -= 1 args: [dynamic]ast.Expr_Id args.allocator = parser.module.allocator skip_newlines(parser) for current(parser).kind != .Right_Paren && current(parser).kind != .Eof { if hole, ok := allow(parser, .Underscore); ok { append(&args, add_expr(parser, ast.Expr{ kind=.Inference_Hole, span=hole.span, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, })) } else { append(&args, parse_expression_bp(parser, 0, nesting+1)) } skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } break } right_paren, ok := allow(parser, .Right_Paren) if !ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after call arguments") right_paren = left_paren } return args[:], right_paren } parse_call :: proc(parser: ^Parser, qualifier: symbol.Id, first, name: token.Token, nesting: int, intrinsic: bool) -> ast.Expr_Id { if nesting >= MAX_EXPRESSION_NESTING { span := skip_parenthesized(parser) return invalid_expr(parser, span, "expression nesting exceeds 256 levels") } args, right_paren := parse_call_args(parser, nesting) return add_expr(parser, ast.Expr{ kind=.Call, span=source.Span{file=name.span.file, start=first.span.start, end=right_paren.span.end}, qualifier=qualifier, name=name.symbol, args=args[:], intrinsic=intrinsic, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_array_literal :: proc(parser: ^Parser, nesting: int) -> ast.Expr_Id { start := advance(parser) parser.delimiter_depth += 1 defer parser.delimiter_depth -= 1 args: [dynamic]ast.Expr_Id args.allocator = parser.module.allocator skip_newlines(parser) for current(parser).kind != .Right_Bracket && current(parser).kind != .Eof { append(&args, parse_expression_bp(parser, 0, nesting+1)) skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } break } end, ok := allow(parser, .Right_Bracket) if !ok { source.add(parser.diagnostics, current(parser).span, "expected ']' after array literal") end = start } return add_expr(parser, ast.Expr{ kind=.Array, span=span_from(start.span, end.span), args=args[:], left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_keyed_initializers :: proc( parser: ^Parser, left_brace: token.Token, nesting: int, close_message: string, ) -> ([]ast.Expr_Id, token.Token) { parser.delimiter_depth += 1 defer parser.delimiter_depth -= 1 args: [dynamic]ast.Expr_Id args.allocator = parser.module.allocator skip_newlines(parser) for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { field, field_ok := parse_member_name(parser) if !field_ok { source.add(parser.diagnostics, field.span, "expected a keyed struct field initializer") break } // A bare key (`T{ variant }`, no `= value`) constructs a void-payload union // variant; the checker validates that the field actually has a void type. value := ast.INVALID_EXPR key_end := field.span if _, ok := allow(parser, .Equal); ok { skip_newlines(parser) value = parse_expression_bp(parser, 0, nesting+1) key_end = parser.module.exprs[value].span } append(&args, add_expr(parser, ast.Expr{ kind=.Keyed, span=span_from(field.span, key_end), name=field.symbol, left=value, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, })) skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } break } right_brace, ok := allow(parser, .Right_Brace) if !ok { source.add(parser.diagnostics, current(parser).span, close_message) right_brace = left_brace } return args[:], right_brace } parse_positional_initializers :: proc( parser: ^Parser, left_brace: token.Token, nesting: int, close_message: string, ) -> ([]ast.Expr_Id, token.Token) { parser.delimiter_depth += 1 defer parser.delimiter_depth -= 1 args: [dynamic]ast.Expr_Id args.allocator = parser.module.allocator skip_newlines(parser) for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { append(&args, parse_expression_bp(parser, 0, nesting+1)) skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } break } right_brace, ok := allow(parser, .Right_Brace) if !ok { source.add(parser.diagnostics, current(parser).span, close_message) right_brace = left_brace } return args[:], right_brace } brace_starts_tuple :: proc(parser: ^Parser) -> bool { if current(parser).kind != .Left_Brace { return false } depth := 0 for cursor := parser.cursor; cursor < len(parser.tokens.items); cursor += 1 { #partial switch parser.tokens.items[cursor].kind { case .Left_Brace, .Left_Paren, .Left_Bracket: depth += 1 case .Right_Brace, .Right_Paren, .Right_Bracket: depth -= 1 if depth == 0 { return cursor == parser.cursor+1 } case .Comma: if depth == 1 { return true } case .Eof: return false case: } } return false } parse_tuple_literal :: proc(parser: ^Parser, nesting: int) -> ast.Expr_Id { left_brace := advance(parser) args, right_brace := parse_positional_initializers(parser, left_brace, nesting, "expected '}' after tuple literal") return add_expr(parser, ast.Expr{ kind=.Struct_Literal, span=span_from(left_brace.span, right_brace.span), args=args, tuple=true, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_struct_literal :: proc( parser: ^Parser, qualifier: symbol.Id, first, name: token.Token, nesting: int, ) -> ast.Expr_Id { left_brace := advance(parser) skip_newlines(parser) keyed_start := (current(parser).kind == .Identifier || token.is_keyword(current(parser).kind)) && (peek(parser).kind == .Equal || peek(parser).kind == .Right_Brace || token.is_keyword(current(parser).kind)) positional := current(parser).kind == .Right_Brace || !keyed_start args: []ast.Expr_Id right_brace: token.Token if positional { args, right_brace = parse_positional_initializers(parser, left_brace, nesting, "expected '}' after tuple literal") } else { args, right_brace = parse_keyed_initializers(parser, left_brace, nesting, "expected '}' after struct literal") } return add_expr(parser, ast.Expr{ kind=.Struct_Literal, span=source.Span{file=name.span.file, start=first.span.start, end=right_brace.span.end}, qualifier=qualifier, name=name.symbol, args=args[:], tuple=positional, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_anonymous_struct_type_expr :: proc(parser: ^Parser) -> ast.Expr_Id { start := advance(parser) fields: [dynamic]types.Field fields.allocator = parser.module.allocator tuple := false if !parse_record_body(parser, &fields, "expected '{' after anonymous struct type", tuple_result=&tuple) { delete(fields) return invalid_expr(parser, start.span, "invalid anonymous struct type") } end := previous(parser) field_start := u32(len(parser.module.type_fields)) field_count := u32(len(fields)) append(&parser.module.type_fields, ..fields[:]) delete(fields) return add_expr(parser, ast.Expr{ kind=.Anonymous_Struct_Type, span=span_from(start.span, end.span), integer=u64(field_start)<<32 | u64(field_count), tuple=tuple, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_integer_magnitude :: proc(text: string) -> (u64, bool) { value: u64 for byte in transmute([]byte)text { if byte < '0' || byte > '9' { return 0, false } next, overflow := intrinsics.overflow_mul(value, u64(10)) if overflow { return 0, false } value, overflow = intrinsics.overflow_add(next, u64(byte-'0')) if overflow { return 0, false } } return value, len(text) > 0 } parse_primary :: proc(parser: ^Parser, nesting: int) -> ast.Expr_Id { tok := current(parser) #partial switch tok.kind { case .Keyword_Int, .Keyword_Uint, .Keyword_Float, .Keyword_Range, .Keyword_Void, .Keyword_Anyopaque, .Keyword_Bool: start := tok target := parse_type_atom(parser) return add_expr(parser, ast.Expr{ kind=.Type, span=start.span, type=target, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Keyword_I8, .Keyword_I16, .Keyword_I32, .Keyword_I64, .Keyword_U8, .Keyword_U16, .Keyword_U32, .Keyword_U64, .Keyword_Isize, .Keyword_Usize, .Keyword_F32, .Keyword_F64, .Keyword_C_Char, .Keyword_C_Schar, .Keyword_C_Uchar, .Keyword_C_Short, .Keyword_C_Ushort, .Keyword_C_Int, .Keyword_C_Uint, .Keyword_C_Long, .Keyword_C_Ulong, .Keyword_C_Longlong, .Keyword_C_Ulonglong, .Keyword_C_Float, .Keyword_C_Double, .Keyword_C_Longdouble: start := tok target := parse_type_atom(parser) if _, ok := allow(parser, .Left_Paren); !ok { return add_expr(parser, ast.Expr{ kind=.Type, span=start.span, type=target, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parser.delimiter_depth += 1 skip_newlines(parser) operand := parse_expression_bp(parser, 0, nesting+1) skip_newlines(parser) end := current(parser) if close, ok := allow(parser, .Right_Paren); ok { end = close } else { source.add(parser.diagnostics, current(parser).span, "expected ')' after scalar cast") } parser.delimiter_depth -= 1 return add_expr(parser, ast.Expr{ kind=.Cast, span=span_from(start.span, end.span), type=target, left=operand, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Question, .At, .Star: start := tok target := parse_type_atom(parser) return add_expr(parser, ast.Expr{ kind=.Type, span=span_from(start.span, previous(parser).span), type=target, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Integer: advance(parser) value, ok := parse_integer_magnitude(token_text(parser, tok)) if !ok { return invalid_expr(parser, tok.span, "integer literal magnitude does not fit in u64") } return add_expr(parser, ast.Expr{ kind=.Integer, span=tok.span, integer=value, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Character: advance(parser) value, ok := decode_character(parser, tok) if !ok { return invalid_expr(parser, tok.span, "character literal must contain one Unicode code point") } return add_expr(parser, ast.Expr{ kind=.Integer, span=tok.span, integer=value, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Float: advance(parser) value, ok := strconv.parse_f64(token_text(parser, tok)) if !ok { return invalid_expr(parser, tok.span, "invalid floating-point literal") } return add_expr(parser, ast.Expr{ kind=.Float, span=tok.span, integer=transmute(u64)value, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .String, .Multiline_String: advance(parser) value := decode_import_path(parser, tok) if tok.kind == .String else decode_multiline_string(parser, tok) id := u64(len(parser.module.strings)) append(&parser.module.strings, value) return add_expr(parser, ast.Expr{ kind=.String, span=tok.span, integer=id, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Keyword_None: advance(parser) return add_expr(parser, ast.Expr{ kind=.None, span=tok.span, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Keyword_Undefined: advance(parser) return add_expr(parser, ast.Expr{ kind=.Undefined, span=tok.span, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Keyword_True, .Keyword_False: advance(parser) return add_expr(parser, ast.Expr{ kind=.Bool, span=tok.span, integer=1 if tok.kind == .Keyword_True else 0, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Keyword_Func: return parse_function_literal(parser) case .Keyword_Struct: return parse_anonymous_struct_type_expr(parser) case .Left_Bracket: if starts_declared_type(parser) { start := tok target := parse_type_atom(parser) return add_expr(parser, ast.Expr{ kind=.Type, span=span_from(start.span, previous(parser).span), type=target, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } return parse_array_literal(parser, nesting) case .Left_Brace: return parse_tuple_literal(parser, nesting) case .Dot: start := advance(parser) member, member_ok := parse_member_name(parser) if !member_ok { return invalid_expr(parser, member.span, "expected an enum member after '.'") } payload := ast.INVALID_EXPR end := member.span if left_brace, ok := allow(parser, .Left_Brace); ok { parser.delimiter_depth += 1 skip_newlines(parser) if current(parser).kind == .Identifier && peek(parser).kind == .Equal { parser.delimiter_depth -= 1 args, right_brace := parse_keyed_initializers(parser, left_brace, nesting, "expected '}' after contextual variant payload") payload = add_expr(parser, ast.Expr{ kind=.Struct_Literal, span=span_from(left_brace.span, right_brace.span), args=args, qualifier=symbol.INVALID, name=symbol.INVALID, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) end = right_brace.span } else if current(parser).kind == .Right_Brace { source.add(parser.diagnostics, current(parser).span, "contextual variant payload requires exactly one expression") right_brace, close_ok := allow(parser, .Right_Brace) if !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '}' after contextual variant payload") right_brace = member } parser.delimiter_depth -= 1 end = right_brace.span } else { payload = parse_expression_bp(parser, 0, nesting+1) skip_newlines(parser) if _, comma_ok := allow(parser, .Comma); comma_ok { source.add(parser.diagnostics, current(parser).span, "contextual variant payload requires exactly one expression") for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { advance(parser) } } right_brace, close_ok := allow(parser, .Right_Brace) if !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '}' after contextual variant payload") right_brace = member } parser.delimiter_depth -= 1 end = right_brace.span } } return add_expr(parser, ast.Expr{ kind=.Enum_Literal, span=span_from(start.span, end), name=member.symbol, left=payload, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Identifier: first := advance(parser) name := first qualifier := symbol.INVALID if current(parser).kind == .Dot && peek(parser).kind != .Integer { advance(parser) member, member_ok := parse_member_name(parser) if !member_ok { return invalid_expr(parser, current(parser).span, "expected a package member after '.'") } qualifier = first.symbol name = member } _, intrinsic := allow(parser, .Bang) if current(parser).kind == .Left_Paren { call := parse_call(parser, qualifier, first, name, nesting, intrinsic) if !intrinsic && current(parser).kind == .Left_Brace && !(parser.no_struct_literal && parser.delimiter_depth == 0) { left_brace := advance(parser) skip_newlines(parser) keyed_start := (current(parser).kind == .Identifier || token.is_keyword(current(parser).kind)) && (peek(parser).kind == .Equal || peek(parser).kind == .Right_Brace || token.is_keyword(current(parser).kind)) positional := current(parser).kind == .Right_Brace || !keyed_start args: []ast.Expr_Id right_brace: token.Token if positional { args, right_brace = parse_positional_initializers(parser, left_brace, nesting, "expected '}' after tuple literal") } else { args, right_brace = parse_keyed_initializers(parser, left_brace, nesting, "expected '}' after struct literal") } return add_expr(parser, ast.Expr{ kind=.Struct_Literal, span=span_from(parser.module.exprs[call].span, right_brace.span), args=args, tuple=positional, left=call, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } return call } if intrinsic { return invalid_expr(parser, previous(parser).span, "expected '(' after intrinsic name") } if current(parser).kind == .Left_Brace && !(parser.no_struct_literal && parser.delimiter_depth == 0) { return parse_struct_literal(parser, qualifier, first, name, nesting) } return add_expr(parser, ast.Expr{ kind=.Name, span=span_from(first.span, name.span), qualifier=qualifier, name=name.symbol, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) case .Underscore: advance(parser) return invalid_expr(parser, tok.span, "'_' is a write-only sink and cannot be read") case .Left_Paren: if nesting >= MAX_EXPRESSION_NESTING { span := skip_parenthesized(parser) return invalid_expr(parser, span, "expression nesting exceeds 256 levels") } advance(parser) parser.delimiter_depth += 1 defer parser.delimiter_depth -= 1 skip_newlines(parser) expr := parse_expression_bp(parser, 0, nesting+1) skip_newlines(parser) if _, ok := allow(parser, .Right_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected ')'") } if expr != ast.INVALID_EXPR && int(expr) < len(parser.module.exprs) { parser.module.exprs[expr].parenthesized = true } return expr case .Invalid: advance(parser) return add_expr(parser, ast.Expr{ kind=.Invalid, span=tok.span, left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=tok.diagnostic, }) } if tok.kind != .Newline && tok.kind != .Right_Brace && tok.kind != .Eof { advance(parser) } return invalid_expr(parser, tok.span, "expected an expression") } infix_binding_power :: proc(kind: token.Kind) -> (left, right: int, ok: bool) { #partial switch kind { case .Range, .Range_Inclusive: return 0, 1, true case .Keyword_Orelse, .Keyword_Catch: return 2, 3, true case .Keyword_Or: return 4, 5, true case .Keyword_And: return 6, 7, true case .Equal_Equal, .Bang_Equal, .Less, .Less_Equal, .Greater, .Greater_Equal: return 8, 9, true case .Plus, .Minus: return 10, 11, true case .Star, .Slash: return 12, 13, true } return 0, 0, false } infix_expr_kind :: proc(kind: token.Kind) -> ast.Expr_Kind { #partial switch kind { case .Range, .Range_Inclusive: return .Range case .Keyword_Orelse: return .Orelse case .Keyword_Catch: return .Catch case .Keyword_Or: return .Or case .Keyword_And: return .And case .Equal_Equal: return .Eq case .Bang_Equal: return .Ne case .Less: return .Lt case .Less_Equal: return .Le case .Greater: return .Gt case .Greater_Equal: return .Ge case .Plus: return .Add case .Minus: return .Sub case .Star: return .Mul case .Slash: return .Div case: return .Add } } compound_assignment_op :: proc(kind: token.Kind) -> (ast.Assignment_Op, bool) { #partial switch kind { case .Plus_Equal: return .Add, true case .Minus_Equal: return .Sub, true case .Star_Equal: return .Mul, true case .Slash_Equal: return .Div, true } return .Set, false } is_simple_range_bound :: proc(expr: ast.Expr) -> bool { if expr.parenthesized { return true } #partial switch expr.kind { case .Integer, .Float, .String, .Bool, .Name: return true case: return false } } prefix_binding_power :: proc(kind: token.Kind) -> (right: int, ok: bool) { #partial switch kind { case .Minus, .Ampersand, .Bang, .Dollar, .Keyword_Try: return 20, true } return 0, false } parse_expression_bp :: proc(parser: ^Parser, minimum_binding_power, nesting: int) -> ast.Expr_Id { if nesting > MAX_EXPRESSION_NESTING { tok := current(parser) if tok.kind != .Newline && tok.kind != .Right_Brace && tok.kind != .Eof { advance(parser) } return invalid_expr(parser, tok.span, "expression nesting exceeds 256 levels") } left := ast.INVALID_EXPR if right_power, ok := prefix_binding_power(current(parser).kind); ok { operator := advance(parser) if operator.kind == .Dollar && current(parser).kind == .Left_Brace { body := parse_block(parser) end := previous(parser) left = add_expr(parser, ast.Expr{ kind = .Comptime, span = span_from(operator.span, end.span), body = body, left = ast.INVALID_EXPR, right = ast.INVALID_EXPR, diagnostic = source.INVALID_DIAGNOSTIC, }) } else { if parser.delimiter_depth > 0 { skip_newlines(parser) } operand := parse_expression_bp(parser, right_power, nesting+1) operand_expr := parser.module.exprs[operand] prefix_kind := ast.Expr_Kind.Negate #partial switch operator.kind { case .Ampersand: prefix_kind = .Address case .Bang: prefix_kind = .Not case .Dollar: prefix_kind = .Comptime case .Keyword_Try: prefix_kind = .Try } left = add_expr(parser, ast.Expr{ kind = prefix_kind, span = span_from(operator.span, operand_expr.span), left = operand, right = ast.INVALID_EXPR, diagnostic = source.INVALID_DIAGNOSTIC, }) } } else { left = parse_primary(parser, nesting) } if parser.delimiter_depth > 0 { skip_newlines(parser) } for { if current(parser).kind == .Caret || current(parser).kind == .Question { operator := advance(parser) left_expr := parser.module.exprs[left] left = add_expr(parser, ast.Expr{ kind=.Deref if operator.kind == .Caret else .Unwrap, span=span_from(left_expr.span, operator.span), left=left, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) continue } if current(parser).kind == .Dot { advance(parser) if current(parser).kind == .Integer { field := advance(parser) text := token_text(parser, field) index, index_ok := parse_integer_magnitude(text) left_expr := parser.module.exprs[left] if !index_ok || (len(text) > 1 && text[0] == '0') { left = invalid_expr(parser, field.span, "tuple field indices must be canonical decimal integers") } else { left = add_expr(parser, ast.Expr{ kind=.Field, span=span_from(left_expr.span, field.span), name=symbol.INVALID, integer=index, left=left, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } continue } field, field_ok := parse_member_name(parser) if !field_ok { left = invalid_expr(parser, field.span, "expected a field name after '.'") continue } left_expr := parser.module.exprs[left] left = add_expr(parser, ast.Expr{ kind=.Field, span=span_from(left_expr.span, field.span), name=field.symbol, left=left, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) continue } if current(parser).kind == .Left_Bracket { start_token := advance(parser) parser.delimiter_depth += 1 skip_newlines(parser) start_expr := ast.INVALID_EXPR end_expr := ast.INVALID_EXPR slicing := false if _, ok := allow(parser, .Range); ok { slicing = true } else { parser.range_disabled += 1 start_expr = parse_expression_bp(parser, 0, nesting+1) parser.range_disabled -= 1 skip_newlines(parser) if _, ok := allow(parser, .Range); ok { slicing = true } } skip_newlines(parser) if slicing && current(parser).kind != .Right_Bracket { parser.range_disabled += 1 end_expr = parse_expression_bp(parser, 0, nesting+1) parser.range_disabled -= 1 skip_newlines(parser) } end_token, ok := allow(parser, .Right_Bracket) if !ok { source.add(parser.diagnostics, current(parser).span, "expected ']' after index or slice") end_token = start_token } parser.delimiter_depth -= 1 left_expr := parser.module.exprs[left] if slicing { args := make([]ast.Expr_Id, 2, parser.module.allocator) args[0] = start_expr args[1] = end_expr left = add_expr(parser, ast.Expr{ kind=.Slice, span=span_from(left_expr.span, end_token.span), args=args, left=left, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } else { left = add_expr(parser, ast.Expr{ kind=.Index, span=span_from(left_expr.span, end_token.span), left=left, right=start_expr, diagnostic=source.INVALID_DIAGNOSTIC, }) } continue } if current(parser).kind == .Bang { marker := advance(parser) if current(parser).kind != .Left_Paren { left = invalid_expr(parser, marker.span, "expected '(' after '!'") continue } left_expr := parser.module.exprs[left] call_span := skip_parenthesized(parser) left = invalid_expr( parser, span_from(left_expr.span, call_span), "intrinsic calls require a direct name", ) continue } if current(parser).kind == .Left_Paren { if nesting >= MAX_EXPRESSION_NESTING { span := skip_parenthesized(parser) left = invalid_expr(parser, span, "expression nesting exceeds 256 levels") continue } left_expr := parser.module.exprs[left] args, right_paren := parse_call_args(parser, nesting) left = add_expr(parser, ast.Expr{ kind=.Call, span=span_from(left_expr.span, right_paren.span), args=args, left=left, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) continue } left_power, right_power, ok := infix_binding_power(current(parser).kind) if ok && (current(parser).kind == .Range || current(parser).kind == .Range_Inclusive) && parser.range_disabled > 0 { ok = false } if !ok || left_power < minimum_binding_power { break } operator := advance(parser) skip_newlines(parser) if operator.kind == .Keyword_Catch { left_expr := parser.module.exprs[left] if _, pipe_ok := allow(parser, .Pipe); pipe_ok { capture := current(parser) if capture.kind != .Identifier && capture.kind != .Underscore { source.add(parser.diagnostics, capture.span, "expected a catch capture name") } else { advance(parser) } if _, close_ok := allow(parser, .Pipe); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '|' after catch capture") } body := parse_block(parser) end := previous(parser) left = add_expr(parser, ast.Expr{ kind=.Catch, span=span_from(left_expr.span, end.span), name=capture.symbol, left=left, right=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) continue } right := parse_expression_bp(parser, right_power, nesting+1) right_expr := parser.module.exprs[right] left = add_expr(parser, ast.Expr{ kind=.Catch, span=span_from(left_expr.span, right_expr.span), left=left, right=right, diagnostic=source.INVALID_DIAGNOSTIC, }) continue } right := parse_expression_bp(parser, right_power, nesting+1) left_expr := parser.module.exprs[left] right_expr := parser.module.exprs[right] if operator.kind == .Range || operator.kind == .Range_Inclusive { if !is_simple_range_bound(left_expr) { source.add( parser.diagnostics, left_expr.span, "range bounds with operators must be parenthesized", ) } if !is_simple_range_bound(right_expr) { source.add( parser.diagnostics, right_expr.span, "range bounds with operators must be parenthesized", ) } } left = add_expr(parser, ast.Expr{ kind=infix_expr_kind(operator.kind), span=span_from(left_expr.span, right_expr.span), integer=1 if operator.kind == .Range_Inclusive else 0, left=left, right=right, diagnostic=source.INVALID_DIAGNOSTIC, }) if parser.delimiter_depth > 0 { skip_newlines(parser) } } return left } parse_expression :: proc(parser: ^Parser) -> ast.Expr_Id { return parse_expression_bp(parser, 0, 0) } finish_statement :: proc(parser: ^Parser, allow_closing_brace := false) -> source.Diagnostic_Id { if current(parser).kind == .Newline { skip_newlines(parser) return source.INVALID_DIAGNOSTIC } if current(parser).kind == .Eof || allow_closing_brace && current(parser).kind == .Right_Brace { return source.INVALID_DIAGNOSTIC } diagnostic := source.add( parser.diagnostics, current(parser).span, "completed statements must be followed by a newline", ) for current(parser).kind != .Newline && current(parser).kind != .Right_Brace && current(parser).kind != .Eof { advance(parser) } skip_newlines(parser) return diagnostic } parse_return :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) if current(parser).kind == .Newline || current(parser).kind == .Right_Brace || current(parser).kind == .Eof { id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Return, span=start.span, expr=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } if cf, is_cf := parse_value_control_flow(parser); is_cf { cf_span := parser.module.statements[cf].span body := make([]ast.Stmt_Id, 1, parser.module.allocator) body[0] = cf id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Return, span=span_from(start.span, cf_span), expr=ast.INVALID_EXPR, body=body, value_control_flow=true, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } expr := parse_expression(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Return, span=span_from(start.span, parser.module.exprs[expr].span), expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // `yield ` supplies a non-void value to an enclosing value construct. // The expression must start on the same line; `break` handles valueless exits. parse_yield :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) // consume 'yield' // `yield :blk x` targets the loop labeled `blk`; a bare `yield x` targets // the directly-enclosing value block / if branch. No expression starts with // ':', so a leading colon is unambiguously a label. label := symbol.INVALID if _, ok := allow(parser, .Colon); ok { if name, name_ok := allow(parser, .Identifier); name_ok { label = name.symbol } else { source.add(parser.diagnostics, current(parser).span, "expected a yield target label after ':'") } } if current(parser).kind == .Newline || current(parser).kind == .Right_Brace || current(parser).kind == .Eof { expr := invalid_expr(parser, start.span, "'yield' must produce a value") id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Yield, span=start.span, label=label, expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } if cf, is_cf := parse_value_control_flow(parser); is_cf { cf_span := parser.module.statements[cf].span body := make([]ast.Stmt_Id, 1, parser.module.allocator) body[0] = cf id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Yield, span=span_from(start.span, cf_span), label=label, expr=ast.INVALID_EXPR, body=body, value_control_flow=true, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } expr := parse_expression(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Yield, span=span_from(start.span, parser.module.exprs[expr].span), label=label, expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // `break` / `continue` carry no value and target the innermost loop; the // checker rejects them outside a loop. parse_loop_control :: proc(parser: ^Parser, kind: ast.Stmt_Kind) -> ast.Stmt_Id { marker := advance(parser) // consume 'break' / 'continue' // `break :outer` / `continue :outer` targets the enclosing loop labeled `outer`. label := symbol.INVALID if _, ok := allow(parser, .Colon); ok { if name, name_ok := allow(parser, .Identifier); name_ok { label = name.symbol } else { source.add(parser.diagnostics, current(parser).span, "expected a loop label after ':'") } } id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=kind, span=marker.span, label=label, expr=ast.INVALID_EXPR, update=ast.INVALID_STMT, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // A bare `{ ... }` introduces a nested scope. Locals declared inside are not // visible after it, and any `defer`s inside it run at the closing brace. A labeled // `blk: { ... }` can be exited early with `break :blk`. parse_block_statement :: proc(parser: ^Parser, label := symbol.INVALID) -> ast.Stmt_Id { start := current(parser).span // the '{' body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Block, span=start, label=label, body=body, expr=ast.INVALID_EXPR, update=ast.INVALID_STMT, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // `defer ` runs whenever the enclosing scope exits; `errdefer` // runs only when it exits through a function error and may capture that error. parse_defer :: proc(parser: ^Parser) -> ast.Stmt_Id { marker := advance(parser) error_only := marker.kind == .Keyword_Errdefer skip_newlines(parser) captures: []symbol.Id if error_only { if _, ok := allow(parser, .Pipe); ok { capture := current(parser) if capture.kind != .Identifier && capture.kind != .Underscore { source.add(parser.diagnostics, capture.span, "expected an errdefer capture name") } else { advance(parser) captures = make([]symbol.Id, 1, parser.module.allocator) captures[0] = capture.symbol } if current(parser).kind == .Comma { source.add(parser.diagnostics, current(parser).span, "'errdefer' accepts exactly one capture") for current(parser).kind != .Pipe && current(parser).kind != .Newline && current(parser).kind != .Eof { advance(parser) } } if _, close_ok := allow(parser, .Pipe); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '|' after errdefer capture") } skip_newlines(parser) } } inner := parse_statement(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Defer, span=marker.span, update=inner, error_only=error_only, captures=captures, expr=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } starts_declared_type :: proc(parser: ^Parser) -> bool { if current(parser).kind != .Left_Bracket { return is_type_token(current(parser).kind) } depth := 0 cursor := parser.cursor for cursor < len(parser.tokens.items) { kind := parser.tokens.items[cursor].kind if kind == .Left_Bracket { depth += 1 } else if kind == .Right_Bracket { depth -= 1 if depth == 0 { cursor += 1 break } } cursor += 1 } if cursor < len(parser.tokens.items) && parser.tokens.items[cursor].kind == .Keyword_Mut { cursor += 1 } return cursor < len(parser.tokens.items) && is_type_token(parser.tokens.items[cursor].kind) } // A declaration/assignment RHS may be a value-producing control-flow construct: // an `if`/`for`/`while` whose branches/iterations `yield`. Returns the parsed // statement (to be carried as a one-element block-init `body`) and true when the // current token opens one. parse_value_control_flow :: proc(parser: ^Parser) -> (ast.Stmt_Id, bool) { #partial switch current(parser).kind { case .Keyword_If: return parse_if(parser), true case .Keyword_For: return parse_for(parser), true case .Keyword_While: return parse_while(parser), true case .Keyword_Match: return parse_match(parser), true } return ast.INVALID_STMT, false } parse_statement :: proc(parser: ^Parser) -> ast.Stmt_Id { if current(parser).kind == .Identifier && token_text(parser, current(parser)) == "inline" && peek(parser).kind == .Keyword_For { start := advance(parser) id := parse_for(parser) parser.module.statements[id].expand = true parser.module.statements[id].span = span_from(start.span, parser.module.statements[id].span) source.add(parser.diagnostics, start.span, "'inline for' was renamed to 'expand for'") return id } if current(parser).kind == .Keyword_Expand && peek(parser).kind == .Keyword_For { start := advance(parser) id := parse_for(parser) parser.module.statements[id].expand = true parser.module.statements[id].span = span_from(start.span, parser.module.statements[id].span) return id } if current(parser).kind == .Keyword_Return { return parse_return(parser) } if current(parser).kind == .Keyword_If { return parse_if(parser) } if current(parser).kind == .Keyword_While { return parse_while(parser) } if current(parser).kind == .Keyword_For { return parse_for(parser) } if current(parser).kind == .Keyword_Break { return parse_loop_control(parser, .Break) } if current(parser).kind == .Keyword_Continue { return parse_loop_control(parser, .Continue) } if current(parser).kind == .Keyword_Defer || current(parser).kind == .Keyword_Errdefer { return parse_defer(parser) } if current(parser).kind == .Keyword_Yield { return parse_yield(parser) } if current(parser).kind == .Keyword_Match { return parse_match(parser) } // A leading `{` opens a bare block scope (struct literals are postfix only). if current(parser).kind == .Left_Brace { return parse_block_statement(parser) } // `blk: { ... }` is a labeled block statement (exitable via `break :blk`). At // statement start, `Identifier ':'` (a single colon, not `::`) opens one. if current(parser).kind == .Identifier && peek(parser).kind == .Colon { label := parse_optional_loop_label(parser) return parse_block_statement(parser, label) } if current(parser).kind == .Identifier || current(parser).kind == .Underscore { start_cursor := parser.cursor name := advance(parser) type_syntax := types.INVALID had_type := false if starts_declared_type(parser) { type_syntax = parse_type(parser) had_type = true } operator := current(parser) if operator.kind == .Colon_Colon || operator.kind == .Equal { advance(parser) skip_newlines(parser) kind := ast.Stmt_Kind.Assignment immutable := false if operator.kind == .Colon_Colon || had_type { kind = .Declaration immutable = operator.kind == .Colon_Colon } // A labeled value block (`x :: blk: { … yield :blk v }`): the label lets a // `yield :blk` exit the block past a nested `if`. Block-init body + label. if current(parser).kind == .Identifier && peek(parser).kind == .Colon { label := parse_optional_loop_label(parser) body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=kind, span=span_from(name.span, previous(parser).span), name=name.symbol, type=type_syntax, immutable=immutable, label=label, target=ast.INVALID_EXPR, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // A `{` on the right is a value block: parse its statements now; the // checker turns its final `yield` into the declared/assigned value. if current(parser).kind == .Left_Brace && !brace_starts_tuple(parser) { brace := current(parser) body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=kind, span=span_from(name.span, brace.span), name=name.symbol, type=type_syntax, immutable=immutable, target=ast.INVALID_EXPR, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // A value-producing `if`/`for`/`while`: carried as a one-element block-init // body, the same signal a value block uses (`expr` invalid). if cf, is_cf := parse_value_control_flow(parser); is_cf { body := make([]ast.Stmt_Id, 1, parser.module.allocator) body[0] = cf id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=kind, span=span_from(name.span, previous(parser).span), name=name.symbol, type=type_syntax, immutable=immutable, value_control_flow=true, target=ast.INVALID_EXPR, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } expr := parse_expression(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=kind, span=span_from(name.span, parser.module.exprs[expr].span), name=name.symbol, type=type_syntax, immutable=immutable, target=ast.INVALID_EXPR, expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } parser.cursor = start_cursor } expr := parse_expression(parser) if _, ok := allow(parser, .Equal); ok { skip_newlines(parser) // A labeled value block assigned to a complex target (`a[i] = blk: { … }`). if current(parser).kind == .Identifier && peek(parser).kind == .Colon { label := parse_optional_loop_label(parser) body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Assignment, span=span_from(parser.module.exprs[expr].span, previous(parser).span), target=expr, label=label, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // A value block assigned to a complex target (`a[i] = { ... }`, `p.f = { ... }`). if current(parser).kind == .Left_Brace && !brace_starts_tuple(parser) { brace := current(parser) body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Assignment, span=span_from(parser.module.exprs[expr].span, brace.span), target=expr, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // A value-producing `if`/`for`/`while` assigned to a complex target. if cf, is_cf := parse_value_control_flow(parser); is_cf { body := make([]ast.Stmt_Id, 1, parser.module.allocator) body[0] = cf id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Assignment, span=span_from(parser.module.exprs[expr].span, previous(parser).span), target=expr, value_control_flow=true, expr=ast.INVALID_EXPR, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } value := parse_expression(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Assignment, span=span_from(parser.module.exprs[expr].span, parser.module.exprs[value].span), target=expr, expr=value, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } if assignment_op, is_compound := compound_assignment_op(current(parser).kind); is_compound { advance(parser) skip_newlines(parser) value := parse_expression(parser) span := span_from(parser.module.exprs[expr].span, parser.module.exprs[value].span) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Assignment, span=span, assignment_op=assignment_op, target=expr, expr=value, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Expression, span=parser.module.exprs[expr].span, expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } parse_params :: proc(parser: ^Parser) -> ([]ast.Param, bool) { params: [dynamic]ast.Param params.allocator = parser.module.allocator variadic := false skip_newlines(parser) for current(parser).kind != .Right_Paren && current(parser).kind != .Eof { if current(parser).kind == .Ellipsis { marker := advance(parser) if variadic { source.add(parser.diagnostics, marker.span, "duplicate variadic marker") } variadic = true skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) } if current(parser).kind != .Right_Paren { source.add(parser.diagnostics, current(parser).span, "variadic marker must be the final parameter") } continue } _, comptime_value := allow(parser, .Dollar) names: [dynamic]token.Token names.allocator = parser.module.allocator for { if current(parser).kind != .Identifier && current(parser).kind != .Underscore { source.add(parser.diagnostics, current(parser).span, "expected parameter name") break } append(&names, advance(parser)) if is_type_token(current(parser).kind) { break } if _, ok := allow(parser, .Comma); !ok { source.add(parser.diagnostics, current(parser).span, "expected ',' or parameter type") break } skip_newlines(parser) } type_syntax := parse_type(parser) for name in names { append(¶ms, ast.Param{ name=name.symbol, span=name.span, type=type_syntax, comptime_value=comptime_value, }) } delete(names) skip_newlines(parser) if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } break } return params[:], variadic } // parse_block parses a brace-delimited statement sequence `{ ... }`, consuming // both braces, and returns the contained statement ids. Shared by function // bodies and control-flow blocks. parse_block :: proc(parser: ^Parser) -> []ast.Stmt_Id { body: [dynamic]ast.Stmt_Id body.allocator = parser.module.allocator if _, ok := allow(parser, .Left_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '{' to open a block") return body[:] } skip_newlines(parser) for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { append(&body, parse_statement(parser)) if diagnostic := finish_statement(parser, true); diagnostic != source.INVALID_DIAGNOSTIC { statement_id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Invalid, span=current(parser).span, expr=ast.INVALID_EXPR, diagnostic=diagnostic, }) append(&body, statement_id) } } if _, ok := allow(parser, .Right_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '}' to close a block") } return body[:] } // parse_control_body parses a braced block (possibly on a following line) or a // single brace-less statement. A non-empty diagnostic enforces the shared // parenthesized-or-call rule for the preceding condition or iterable. parse_control_body :: proc(parser: ^Parser, header: ast.Expr_Id, diagnostic: string) -> []ast.Stmt_Id { skip_newlines(parser) if current(parser).kind == .Left_Brace { return parse_block(parser) } if len(diagnostic) > 0 && header != ast.INVALID_EXPR && int(header) < len(parser.module.exprs) { expr := parser.module.exprs[header] if !expr.parenthesized && expr.kind != .Call { source.add(parser.diagnostics, expr.span, diagnostic) } } single := make([]ast.Stmt_Id, 1, parser.module.allocator) single[0] = parse_statement(parser) return single } parse_if :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) // consume 'if' skip_newlines(parser) saved := parser.no_struct_literal parser.no_struct_literal = true condition := parse_expression(parser) parser.no_struct_literal = saved captures: [dynamic]symbol.Id captures.allocator = parser.module.allocator guard := ast.INVALID_EXPR if _, ok := allow(parser, .Pipe); ok { for { name_tok := current(parser) if name_tok.kind == .Identifier || name_tok.kind == .Underscore { advance(parser) append(&captures, name_tok.symbol) } else { source.add(parser.diagnostics, current(parser).span, "expected an unwrap capture name") break } if _, comma_ok := allow(parser, .Comma); !comma_ok { break } if current(parser).kind == .Colon || current(parser).kind == .Pipe { source.add(parser.diagnostics, current(parser).span, "expected an unwrap capture after ','") break } } if _, guard_ok := allow(parser, .Colon); guard_ok { if current(parser).kind == .Pipe { source.add(parser.diagnostics, current(parser).span, "expected a guard expression after ':'") } else { saved = parser.no_struct_literal parser.no_struct_literal = true guard = parse_expression(parser) parser.no_struct_literal = saved } } if _, close_ok := allow(parser, .Pipe); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '|' to close unwrap captures") } } then_body := parse_control_body( parser, condition, "a brace-less 'if' body requires the condition to be parenthesized unless it is a function call", ) else_body: []ast.Stmt_Id = nil saved_cursor := parser.cursor skip_newlines(parser) if current(parser).kind == .Keyword_Else { advance(parser) skip_newlines(parser) if current(parser).kind == .Keyword_If { nested := parse_if(parser) single := make([]ast.Stmt_Id, 1, parser.module.allocator) single[0] = nested else_body = single } else { else_body = parse_control_body(parser, ast.INVALID_EXPR, "") } } else { parser.cursor = saved_cursor } id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.If, span=span_from(start.span, previous(parser).span), expr=condition, captures=captures[:], guard=guard, body=then_body, else_body=else_body, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // parse_arm_body parses a match arm's body after the `:`: a braced block (whose // inner statements are returned unwrapped, like `parse_control_body`) or a single // brace-less statement. For value-match a brace-less body is a single expression // that the checker yields implicitly. parse_arm_body :: proc(parser: ^Parser) -> []ast.Stmt_Id { skip_newlines(parser) if current(parser).kind == .Left_Brace { return parse_block(parser) } single := make([]ast.Stmt_Id, 1, parser.module.allocator) single[0] = parse_statement(parser) return single } // parse_match_arm parses one ` [|[@]capture|]: `, `else: `, // or `expand |[@]value[, tag]|: ` arm. parse_match_arm :: proc(parser: ^Parser) -> ast.Stmt_Id { start := current(parser).span patterns: [dynamic]ast.Expr_Id patterns.allocator = parser.module.allocator captures: [dynamic]symbol.Id captures.allocator = parser.module.allocator pointer_capture := false expand := false if _, is_expand := allow(parser, .Keyword_Expand); is_expand { expand = true if _, ok := allow(parser, .Pipe); !ok { source.add(parser.diagnostics, current(parser).span, "expected '|' before expand captures") } else { if _, at_ok := allow(parser, .At); at_ok { pointer_capture = true } name_tok := current(parser) if name_tok.kind == .Identifier || name_tok.kind == .Underscore { advance(parser) append(&captures, name_tok.symbol) } else { source.add(parser.diagnostics, current(parser).span, "expected an expand value capture") } if _, comma_ok := allow(parser, .Comma); comma_ok { tag_tok := current(parser) if tag_tok.kind == .Identifier || tag_tok.kind == .Underscore { advance(parser) append(&captures, tag_tok.symbol) } else { source.add(parser.diagnostics, current(parser).span, "expected an expand tag capture") } if _, extra := allow(parser, .Comma); extra { source.add(parser.diagnostics, current(parser).span, "'expand' accepts at most two captures") for current(parser).kind != .Pipe && current(parser).kind != .Colon && current(parser).kind != .Newline && current(parser).kind != .Eof { advance(parser) } } } if _, close_ok := allow(parser, .Pipe); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '|' to close expand captures") } } } else if _, is_else := allow(parser, .Keyword_Else); !is_else { saved := parser.no_struct_literal parser.no_struct_literal = true append(&patterns, parse_expression(parser)) for { if _, ok := allow(parser, .Comma); !ok { break } skip_newlines(parser) append(&patterns, parse_expression(parser)) } parser.no_struct_literal = saved if _, ok := allow(parser, .Pipe); ok { if _, at_ok := allow(parser, .At); at_ok { pointer_capture = true } name_tok := current(parser) if name_tok.kind == .Identifier || name_tok.kind == .Underscore { advance(parser) append(&captures, name_tok.symbol) } else { source.add(parser.diagnostics, current(parser).span, "expected a capture name after '|'") } if _, close_ok := allow(parser, .Pipe); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected '|' to close the match capture") } } } if _, ok := allow(parser, .Colon); !ok { source.add(parser.diagnostics, current(parser).span, "expected ':' after a match pattern") } body := parse_arm_body(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Match_Arm, span=span_from(start, previous(parser).span), expr=ast.INVALID_EXPR, patterns=patterns[:], captures=captures[:], pointer_capture=pointer_capture, expand=expand, body=body, target=ast.INVALID_EXPR, update=ast.INVALID_STMT, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } // parse_match parses `match { * }`. Arms are newline-separated; each // is a `Match_Arm` statement stored in the `Match`'s `body`. Usable as a statement // and (via `parse_value_control_flow`) as a value source on a declaration/assignment. parse_match :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) // consume 'match' skip_newlines(parser) saved := parser.no_struct_literal parser.no_struct_literal = true subject := parse_expression(parser) parser.no_struct_literal = saved arms: [dynamic]ast.Stmt_Id arms.allocator = parser.module.allocator skip_newlines(parser) if _, ok := allow(parser, .Left_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '{' to open match arms") } skip_newlines(parser) for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { append(&arms, parse_match_arm(parser)) if diagnostic := finish_statement(parser, true); diagnostic != source.INVALID_DIAGNOSTIC { arm_id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Invalid, span=current(parser).span, expr=ast.INVALID_EXPR, diagnostic=diagnostic, }) append(&arms, arm_id) } } if _, ok := allow(parser, .Right_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '}' to close match arms") } id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Match, span=span_from(start.span, previous(parser).span), expr=subject, body=arms[:], target=ast.INVALID_EXPR, update=ast.INVALID_STMT, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } parse_while_update :: proc(parser: ^Parser) -> ast.Stmt_Id { parenthesized := false if _, ok := allow(parser, .Left_Paren); ok { parenthesized = true parser.delimiter_depth += 1 skip_newlines(parser) } update := ast.INVALID_STMT if current(parser).kind == .Left_Brace || current(parser).kind == .Right_Paren || current(parser).kind == .Eof { diagnostic := source.add( parser.diagnostics, current(parser).span, "expected a while update statement after ':'", ) update = ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.Invalid, span=current(parser).span, expr=ast.INVALID_EXPR, update=ast.INVALID_STMT, diagnostic=diagnostic, }) } else { saved := parser.no_struct_literal parser.no_struct_literal = true update = parse_statement(parser) parser.no_struct_literal = saved statement := &parser.module.statements[update] switch statement.kind { case .Assignment, .Expression: case .Invalid, .Declaration, .Return, .If, .While, .For, .Break, .Continue, .Block, .Defer, .Yield, .Match, .Match_Arm: diagnostic := source.add( parser.diagnostics, statement.span, "while update must be an assignment, sink, or expression statement", ) statement.kind = .Invalid statement.diagnostic = diagnostic } } if parenthesized { skip_newlines(parser) if _, ok := allow(parser, .Right_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after while update") for current(parser).kind != .Right_Paren && current(parser).kind != .Left_Brace && current(parser).kind != .Newline && current(parser).kind != .Eof { advance(parser) } _, _ = allow(parser, .Right_Paren) } parser.delimiter_depth -= 1 } return update } parse_for :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) // consume 'for' skip_newlines(parser) saved := parser.no_struct_literal parser.no_struct_literal = true iterable := parse_expression(parser) parser.no_struct_literal = saved skip_newlines(parser) pointer_capture := false item_name := symbol.INVALID index_name := symbol.INVALID if _, ok := allow(parser, .Pipe); !ok { source.add(parser.diagnostics, current(parser).span, "expected '|' before for-loop captures") } else { if _, ok := allow(parser, .At); ok { pointer_capture = true } item, item_ok := allow(parser, .Identifier) if item_ok { item_name = item.symbol } else { source.add(parser.diagnostics, current(parser).span, "expected a for-loop item capture") } if _, ok := allow(parser, .Comma); ok { index, index_ok := allow(parser, .Identifier) if index_ok { index_name = index.symbol } else { source.add(parser.diagnostics, current(parser).span, "expected an index capture after ','") } } if _, ok := allow(parser, .Pipe); !ok { source.add(parser.diagnostics, current(parser).span, "expected '|' to close for-loop captures") } } skip_newlines(parser) label := parse_optional_loop_label(parser) body := parse_control_body( parser, iterable, "a brace-less 'for' body requires the iterable to be parenthesized unless it is a function call", ) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.For, span=span_from(start.span, previous(parser).span), name=item_name, index_name=index_name, label=label, pointer_capture=pointer_capture, expr=iterable, body=body, update=ast.INVALID_STMT, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } parse_while :: proc(parser: ^Parser) -> ast.Stmt_Id { start := advance(parser) // consume 'while' skip_newlines(parser) saved := parser.no_struct_literal parser.no_struct_literal = true condition := parse_expression(parser) parser.no_struct_literal = saved skip_newlines(parser) update := ast.INVALID_STMT if _, ok := allow(parser, .Colon); ok { skip_newlines(parser) update = parse_while_update(parser) skip_newlines(parser) } label := parse_optional_loop_label(parser) body := parse_block(parser) id := ast.stmt_id(len(parser.module.statements)) append(&parser.module.statements, ast.Stmt{ kind=.While, span=span_from(start.span, previous(parser).span), expr=condition, body=body, label=label, update=update, diagnostic=source.INVALID_DIAGNOSTIC, }) return id } parse_function :: proc(parser: ^Parser, name: token.Token, c_abi, file_hidden: bool) { advance(parser) if _, ok := allow(parser, .Left_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected '(' after 'func'") } params, variadic := parse_params(parser) if _, ok := allow(parser, .Right_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after parameters") } skip_newlines(parser) result := parse_type(parser) error_type := types.INVALID if _, ok := allow(parser, .Bang); ok { error_type = parse_error_type(parser) } end := previous(parser) ended_by_newline := current(parser).kind == .Newline if current(parser).kind == .Newline { skip_newlines(parser) } if current(parser).kind != .Left_Brace { if !ended_by_newline && current(parser).kind != .Eof { _ = finish_statement(parser) } _ = ast.function_id(len(parser.module.functions)) append(&parser.module.functions, ast.Function{ span=span_from(name.span, end.span), name=name.symbol, pkg=parser.pkg, file=parser.file, c_abi=c_abi, file_hidden=file_hidden, has_body=false, variadic=variadic, params=params, result=result, error=error_type, diagnostic=source.INVALID_DIAGNOSTIC, }) return } body := parse_block(parser) end = previous(parser) _ = ast.function_id(len(parser.module.functions)) append(&parser.module.functions, ast.Function{ span=span_from(name.span, end.span), name=name.symbol, pkg=parser.pkg, file=parser.file, c_abi=c_abi, file_hidden=file_hidden, has_body=true, variadic=variadic, params=params, result=result, error=error_type, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_function_literal :: proc(parser: ^Parser) -> ast.Expr_Id { start := advance(parser) if _, ok := allow(parser, .Left_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected '(' after 'func'") } params, variadic := parse_params(parser) if _, ok := allow(parser, .Right_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after parameters") } skip_newlines(parser) result := parse_type(parser) error_type := types.INVALID if _, ok := allow(parser, .Bang); ok { error_type = parse_error_type(parser) } if current(parser).kind == .Newline { skip_newlines(parser) } if current(parser).kind != .Left_Brace { delete(params, parser.module.allocator) return invalid_expr(parser, start.span, "expected function literal body") } body := parse_block(parser) end := previous(parser) function_id := ast.function_id(len(parser.module.functions)) append(&parser.module.functions, ast.Function{ span=span_from(start.span, end.span), name=symbol.INVALID, pkg=parser.pkg, file=parser.file, c_abi=false, generated=true, has_body=true, variadic=variadic, params=params, result=result, error=error_type, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) return add_expr(parser, ast.Expr{ kind=.Function_Literal, span=span_from(start.span, end.span), integer=u64(function_id), left=ast.INVALID_EXPR, right=ast.INVALID_EXPR, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_record_field_type :: proc(parser: ^Parser, allow_anonymous_struct_payload: bool) -> types.Type { if allow_anonymous_struct_payload && current(parser).kind == .Keyword_Struct { return parse_inline_struct_payload_type(parser) } return parse_type(parser) } parse_record_body :: proc( parser: ^Parser, fields: ^[dynamic]types.Field, expected_open: string, allow_anonymous_struct_payload := false, allow_keyword_names := false, tuple_result: ^bool = nil, ) -> bool { if _, ok := allow(parser, .Left_Brace); !ok { source.add(parser.diagnostics, current(parser).span, expected_open) return false } skip_newlines(parser) mode := 0 // 0 unknown, 1 named, 2 tuple for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { unnamed := false if !allow_keyword_names { if current(parser).kind == .Identifier { next := peek(parser).kind unnamed = next == .Comma || next == .Newline || next == .Right_Brace || next == .Dot || next == .Left_Paren } else { unnamed = is_type_token(current(parser).kind) } } if unnamed { if mode == 1 { source.add(parser.diagnostics, current(parser).span, "struct fields cannot mix named and unnamed forms") } mode = 2 field_type := parse_type(parser) append(fields, types.Field{name=0, type=field_type}) } else { if mode == 2 { source.add(parser.diagnostics, current(parser).span, "struct fields cannot mix named and unnamed forms") } mode = 1 field_name, field_ok := parse_member_name(parser, allow_keyword_names) if !field_ok { source.add(parser.diagnostics, current(parser).span, "expected a struct field name or tuple element type") for current(parser).kind != .Newline && current(parser).kind != .Right_Brace && current(parser).kind != .Eof { advance(parser) } skip_newlines(parser) continue } field_type := parse_record_field_type(parser, allow_anonymous_struct_payload) append(fields, types.Field{name=u32(field_name.symbol), type=field_type}) } if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) continue } _ = finish_statement(parser, true) } if _, ok := allow(parser, .Right_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '}' after struct fields") } if tuple_result != nil { tuple_result^ = mode == 2 } return true } parse_inline_struct_payload_type :: proc(parser: ^Parser) -> types.Type { advance(parser) fields: [dynamic]types.Field fields.allocator = parser.module.allocator defer delete(fields) if !parse_record_body(parser, &fields, "expected '{' after anonymous struct payload") { return types.INVALID } return types.struct_anonymous(&parser.module.type_store, fields[:]) } parse_inline_union_type :: proc(parser: ^Parser) -> types.Type { advance(parser) valid := true if _, ok := allow(parser, .Left_Paren); !ok { source.add(parser.diagnostics, current(parser).span, "expected '(enum)' after inline union error type") valid = false } else { if _, enum_ok := allow(parser, .Keyword_Enum); !enum_ok { source.add(parser.diagnostics, current(parser).span, "inline union error types must use 'union(enum)'") if current(parser).kind != .Right_Paren { _ = parse_type(parser) } valid = false } if _, close_ok := allow(parser, .Right_Paren); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after union tag") valid = false } } fields: [dynamic]types.Field fields.allocator = parser.module.allocator defer delete(fields) if !parse_record_body(parser, &fields, "expected '{' after inline union error type", true, true) || !valid { return types.INVALID } tag := synthesize_union_tag(parser, fields[:]) return types.union_anonymous(&parser.module.type_store, fields[:], tag) } parse_struct :: proc(parser: ^Parser, name: token.Token, c_layout, file_hidden: bool, is_union := false) { start := advance(parser) id := types.named(&parser.module.type_store, u32(parser.pkg), u32(name.symbol), file=u32(parser.file), file_hidden=file_hidden) // A tagged union spells its discriminant in parens: `union(Enum)` reuses an existing // enum; `union(enum)` synthesizes one from the variant names after the body is parsed. tag := types.INVALID declared_tag := types.INVALID inferred_tag := false if is_union { if _, ok := allow(parser, .Left_Paren); ok { if _, enum_ok := allow(parser, .Keyword_Enum); enum_ok { inferred_tag = true } else { declared_tag = parse_type(parser) } if _, close_ok := allow(parser, .Right_Paren); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after union tag") } } } ended_by_newline := current(parser).kind == .Newline skip_newlines(parser) if current(parser).kind != .Left_Brace { if c_layout { source.add(parser.diagnostics, start.span, "c_struct declarations require a body; use 'opaque' for incomplete types") if !ended_by_newline { _ = finish_statement(parser) } return } if !c_layout { source.add( parser.diagnostics, start.span, "native union declarations require a body" if is_union else "native struct declarations require a body", ) } if !types.define_record(&parser.module.type_store, id, nil, c_layout, true, is_union, tag=tag, declared_tag=declared_tag) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } if !ended_by_newline { _ = finish_statement(parser) } return } fields: [dynamic]types.Field fields.allocator = parser.module.allocator defer delete(fields) allow_anonymous_struct_payload := is_union && (inferred_tag || types.is_valid(declared_tag)) tuple := false _ = parse_record_body( parser, &fields, "expected '{' after struct fields", allow_anonymous_struct_payload, allow_anonymous_struct_payload, &tuple, ) if tuple && (c_layout || is_union) { source.add(parser.diagnostics, start.span, "unnamed fields are only supported by native structs") tuple = false } if is_union && (inferred_tag || types.is_valid(declared_tag)) { tag = synthesize_union_tag(parser, fields[:]) } if !types.define_record(&parser.module.type_store, id, fields[:], c_layout, false, is_union, tag=tag, declared_tag=declared_tag, tuple=tuple) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } _ = finish_statement(parser) } parse_opaque :: proc(parser: ^Parser, name: token.Token, file_hidden: bool) { start := advance(parser) id := types.named(&parser.module.type_store, u32(parser.pkg), u32(name.symbol), file=u32(parser.file), file_hidden=file_hidden) if !types.define_record(&parser.module.type_store, id, nil, false, true, false) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } if current(parser).kind == .Left_Brace { source.add(parser.diagnostics, start.span, "opaque declarations do not have a body") } _ = finish_statement(parser) } // synthesize_union_tag builds the anonymous runtime discriminant enum for a tagged // union: one member per variant, valued by the program-global (name, payload-type) // ID. The declared tag enum, if any, remains only the validation surface. synthesize_union_tag :: proc(parser: ^Parser, fields: []types.Field) -> types.Type { members := make([]types.Enum_Member, len(fields), parser.module.allocator) defer delete(members, parser.module.allocator) for field, index in fields { id, ok := types.variant_id(&parser.module.type_store, field.name, field.type) if !ok { source.add(parser.diagnostics, current(parser).span, "too many global sum variants for u16 tags") } members[index] = types.Enum_Member{name=field.name, value=i128(id)} } return types.enum_anonymous(&parser.module.type_store, members, types.U16) } parse_distinct :: proc(parser: ^Parser, name: token.Token, file_hidden: bool) { start := advance(parser) child := parse_type(parser) id := types.named(&parser.module.type_store, u32(parser.pkg), u32(name.symbol), file=u32(parser.file), file_hidden=file_hidden) if !types.define_distinct(&parser.module.type_store, id, child) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } if !types.is_valid(child) { source.add(parser.diagnostics, start.span, "distinct declarations require a backing type") } _ = finish_statement(parser) } parse_alias :: proc(parser: ^Parser, name: token.Token, file_hidden: bool) { start := advance(parser) saved := parser.cursor if current(parser).kind == .Identifier && peek(parser).kind == .Dot { qualifier := advance(parser) advance(parser) if current(parser).kind == .Identifier { member := advance(parser) if current(parser).kind == .Newline || current(parser).kind == .Eof { append(&parser.module.aliases, ast.Declaration_Alias{ span=span_from(name.span, member.span), name=name.symbol, qualifier=qualifier.symbol, member=member.symbol, pkg=parser.pkg, file=parser.file, target_pkg=ast.INVALID_PACKAGE, file_hidden=file_hidden, valid=true, diagnostic=source.INVALID_DIAGNOSTIC, }) _ = finish_statement(parser) return } } } parser.cursor = saved child := parse_type(parser) id := types.named(&parser.module.type_store, u32(parser.pkg), u32(name.symbol), file=u32(parser.file), file_hidden=file_hidden) if !types.define_alias(&parser.module.type_store, id, child) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } if !types.is_valid(child) { source.add(parser.diagnostics, start.span, "alias declarations require a backing type") } _ = finish_statement(parser) } parse_enum_body :: proc( parser: ^Parser, start: source.Span, explicit_backing: bool, backing: ^types.Type, members: ^[dynamic]types.Enum_Member, values: ^[dynamic]ast.Enum_Value, deferred: ^bool, expected_open: string, ) -> bool { skip_newlines(parser) if _, ok := allow(parser, .Left_Brace); !ok { source.add(parser.diagnostics, current(parser).span, expected_open) return false } next_value: i128 previous_value: i128 has_previous := false order_known := true skip_newlines(parser) for current(parser).kind != .Right_Brace && current(parser).kind != .Eof { member, member_ok := parse_member_name(parser) if !member_ok { source.add(parser.diagnostics, current(parser).span, "expected an enum member name") for current(parser).kind != .Newline && current(parser).kind != .Right_Brace && current(parser).kind != .Eof { advance(parser) } skip_newlines(parser) continue } duplicate := false for existing in members^ { if existing.name == u32(member.symbol) { duplicate = true break } } if duplicate { source.addf(parser.diagnostics, member.span, "duplicate enum member '%s'", token_text(parser, member)) } value := next_value explicit := false value_expr := ast.INVALID_EXPR if _, ok := allow(parser, .Equal); ok { explicit = true if !explicit_backing { source.add(parser.diagnostics, member.span, "explicit enum values require a backing type") } saved := parser.cursor negative := false if _, minus_ok := allow(parser, .Minus); minus_ok { negative = true } literal := current(parser) if literal.kind == .Integer { advance(parser) magnitude, magnitude_ok := parse_integer_magnitude(token_text(parser, literal)) if !magnitude_ok { source.add(parser.diagnostics, literal.span, "enum value magnitude does not fit in u64") } else { value = i128(magnitude) if negative { value = -value } } } else { parser.cursor = saved value_expr = parse_expression(parser) deferred^ = true order_known = false } } if order_known && has_previous && value <= previous_value { source.add(parser.diagnostics, member.span, "enum values must be strictly increasing") } if !duplicate { append(members, types.Enum_Member{name=u32(member.symbol), value=value}) append(values, ast.Enum_Value{expr=value_expr, span=member.span, explicit=explicit}) } previous_value = value has_previous = true next_value = value+1 if _, ok := allow(parser, .Comma); ok { skip_newlines(parser) } else { _ = finish_statement(parser, true) } } if _, ok := allow(parser, .Right_Brace); !ok { source.add(parser.diagnostics, current(parser).span, "expected '}' after enum members") } if len(members^) == 0 { source.add(parser.diagnostics, start, "enum declarations require at least one member") } if !explicit_backing { backing^ = types.U16 for &member in members^ { id, ok := types.variant_id(&parser.module.type_store, member.name, types.VOID) if !ok { source.add(parser.diagnostics, start, "too many global sum variants for u16 tags") } member.value = i128(id) } } return true } parse_inline_enum_type :: proc(parser: ^Parser) -> types.Type { start := advance(parser) valid := true if _, ok := allow(parser, .Left_Paren); ok { source.add(parser.diagnostics, start.span, "inline enum error types cannot declare a backing type") _ = parse_type(parser) if _, close_ok := allow(parser, .Right_Paren); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after enum backing type") } valid = false } backing := types.INVALID members: [dynamic]types.Enum_Member members.allocator = parser.module.allocator defer delete(members) values: [dynamic]ast.Enum_Value values.allocator = parser.module.allocator defer delete(values) deferred := false if !parse_enum_body(parser, start.span, false, &backing, &members, &values, &deferred, "expected '{' after inline enum error type") || !valid { return types.INVALID } return types.enum_anonymous(&parser.module.type_store, members[:], backing) } parse_enum :: proc(parser: ^Parser, name: token.Token, file_hidden: bool) { start := advance(parser) explicit_backing := false backing := types.INVALID if _, ok := allow(parser, .Left_Paren); ok { explicit_backing = true backing = parse_type(parser) if _, close_ok := allow(parser, .Right_Paren); !close_ok { source.add(parser.diagnostics, current(parser).span, "expected ')' after enum backing type") } } members: [dynamic]types.Enum_Member members.allocator = parser.module.allocator defer delete(members) values: [dynamic]ast.Enum_Value values.allocator = parser.module.allocator defer delete(values) deferred := false if !parse_enum_body(parser, start.span, explicit_backing, &backing, &members, &values, &deferred, "expected '{' after enum declaration") { _ = finish_statement(parser) return } id := types.named(&parser.module.type_store, u32(parser.pkg), u32(name.symbol), file=u32(parser.file), file_hidden=file_hidden) if !types.define_enum(&parser.module.type_store, id, backing, members[:], explicit_backing) { source.addf(parser.diagnostics, name.span, "duplicate type declaration '%s'", token_text(parser, name)) } if explicit_backing && deferred { stored := make([]ast.Enum_Value, len(values), parser.module.allocator) copy(stored, values[:]) append(&parser.module.enum_declarations, ast.Enum_Declaration{ type=id, pkg=parser.pkg, file=parser.file, values=stored, }) } _ = finish_statement(parser) } decode_import_path :: proc(parser: ^Parser, tok: token.Token) -> string { text := token_text(parser, tok) if len(text) < 2 { return fmt.aprintf("", allocator=parser.module.allocator) } builder := strings.builder_make(parser.module.allocator) defer strings.builder_destroy(&builder) for index := 1; index < len(text)-1; index += 1 { value := text[index] if value == '\\' && index+1 < len(text)-1 { index += 1 value = text[index] switch value { case 'n': value = '\n' case 'r': value = '\r' case 't': value = '\t' case '0': value = 0 case: } } strings.write_byte(&builder, value) } return fmt.aprintf("%s", strings.to_string(builder), allocator=parser.module.allocator) } decode_multiline_string :: proc(parser: ^Parser, tok: token.Token) -> string { text := token_text(parser, tok) builder := strings.builder_make(parser.module.allocator) defer strings.builder_destroy(&builder) first := true index := 0 for index < len(text) { for index < len(text) && (text[index] == ' ' || text[index] == '\t') { index += 1 } if index >= len(text) || text[index] != '`' { break } index += 1 // skip backtick if !first { strings.write_byte(&builder, '\n') } first = false for index < len(text) && text[index] != '\n' { strings.write_byte(&builder, text[index]) index += 1 } index += 1 // skip newline (terminates this line) } return fmt.aprintf("%s", strings.to_string(builder), allocator=parser.module.allocator) } parse_import :: proc(parser: ^Parser, alias: token.Token, start: token.Token, test_only := false) { skip_newlines(parser) path_token := current(parser) if path_token.kind != .String { source.add(parser.diagnostics, path_token.span, "expected an import path string") if path_token.kind != .Newline && path_token.kind != .Eof { advance(parser) } _ = ast.import_id(len(parser.module.imports)) append(&parser.module.imports, ast.Import{ span=start.span, alias=alias.symbol, pkg=parser.pkg, file=parser.file, target=ast.INVALID_PACKAGE, valid=false, test_only=test_only, diagnostic=source.INVALID_DIAGNOSTIC, }) _ = finish_statement(parser) return } advance(parser) _ = ast.import_id(len(parser.module.imports)) append(&parser.module.imports, ast.Import{ span=span_from(start.span, path_token.span), alias=alias.symbol, path=decode_import_path(parser, path_token), pkg=parser.pkg, file=parser.file, target=ast.INVALID_PACKAGE, valid=true, test_only=test_only, diagnostic=source.INVALID_DIAGNOSTIC, }) _ = finish_statement(parser) } parse_test :: proc(parser: ^Parser, name: token.Token) { advance(parser) // consume 'test' skip_newlines(parser) if current(parser).kind != .Left_Brace { source.add(parser.diagnostics, current(parser).span, "expected '{' after test name") _ = finish_statement(parser) return } body := parse_block(parser) append(&parser.module.functions, ast.Function{ span=span_from(name.span, previous(parser).span), name=name.symbol, pkg=parser.pkg, file=parser.file, test=true, has_body=true, result=types.VOID, body=body, diagnostic=source.INVALID_DIAGNOSTIC, }) } parse_top_level :: proc(parser: ^Parser) { hide_token, file_hidden := allow(parser, .Keyword_Hide) if current(parser).kind == .Keyword_Test && peek(parser).kind == .Keyword_Import { if file_hidden { source.add(parser.diagnostics, hide_token.span, "test imports cannot use 'hide'") } start := advance(parser) advance(parser) // consume 'import' parse_import(parser, token.Token{}, start, test_only=true) return } if current(parser).kind == .Keyword_Import { if file_hidden { source.add(parser.diagnostics, hide_token.span, "imports are already file-local and cannot use 'hide'") } start := advance(parser) parse_import(parser, token.Token{}, start) return } if current(parser).kind != .Identifier { message := "expected a declaration name after 'hide'" if file_hidden else "expected a top-level declaration" source.add(parser.diagnostics, current(parser).span, message) for current(parser).kind != .Newline && current(parser).kind != .Eof { advance(parser) } _ = finish_statement(parser) return } name := advance(parser) if current(parser).kind == .Keyword_Test { if file_hidden { source.add(parser.diagnostics, hide_token.span, "test declarations cannot use 'hide'") } parse_test(parser, name) return } if current(parser).kind == .Colon_Colon { saved := parser.cursor advance(parser) skip_newlines(parser) if current(parser).kind == .Keyword_Import { if file_hidden { source.add(parser.diagnostics, hide_token.span, "imports are already file-local and cannot use 'hide'") } start := advance(parser) parse_import(parser, name, start) return } parser.cursor = saved } if current(parser).kind == .Keyword_Func || current(parser).kind == .Keyword_C_Func { c_abi := current(parser).kind == .Keyword_C_Func parse_function(parser, name, c_abi, file_hidden) return } type_syntax := types.INVALID if is_type_token(current(parser).kind) { type_syntax = parse_type(parser) } operator := current(parser) if operator.kind != .Colon_Colon && operator.kind != .Equal { source.add(parser.diagnostics, operator.span, "expected '::' or '=' after top-level name") _ = finish_statement(parser) return } advance(parser) skip_newlines(parser) if operator.kind == .Colon_Colon && (current(parser).kind == .Keyword_Func || current(parser).kind == .Keyword_C_Func) { source.add(parser.diagnostics, span_from(name.span, current(parser).span), "function declarations do not use '::'; write 'name func(...)' or 'name c_func(...)'") c_abi := current(parser).kind == .Keyword_C_Func parse_function(parser, name, c_abi, file_hidden) return } if operator.kind == .Colon_Colon && (current(parser).kind == .Keyword_Struct || current(parser).kind == .Keyword_C_Struct) { parse_struct(parser, name, current(parser).kind == .Keyword_C_Struct, file_hidden) return } if operator.kind == .Colon_Colon && current(parser).kind == .Keyword_Opaque { parse_opaque(parser, name, file_hidden) return } if operator.kind == .Colon_Colon && current(parser).kind == .Keyword_Union { parse_struct(parser, name, false, file_hidden, is_union=true) return } if operator.kind == .Colon_Colon && current(parser).kind == .Keyword_Enum { parse_enum(parser, name, file_hidden) return } if operator.kind == .Colon_Colon && current(parser).kind == .Keyword_Distinct { parse_distinct(parser, name, file_hidden) return } if operator.kind == .Colon_Colon && current(parser).kind == .Keyword_Alias { parse_alias(parser, name, file_hidden) return } expr := parse_expression(parser) _ = ast.global_id(len(parser.module.globals)) append(&parser.module.globals, ast.Global{ span=span_from(name.span, parser.module.exprs[expr].span), name=name.symbol, pkg=parser.pkg, file=parser.file, file_hidden=file_hidden, type=type_syntax, immutable=operator.kind == .Colon_Colon, expr=expr, diagnostic=source.INVALID_DIAGNOSTIC, }) _ = finish_statement(parser) } parse :: proc( stream: ^token.Stream, source_file: ^source.Source, diagnostics: ^source.Diagnostics, allocator := context.allocator, ) -> ast.Module { parser := Parser{ tokens=stream, source_file=source_file, diagnostics=diagnostics, module=ast.init_module(allocator), } parser.hidden_names.allocator = allocator defer delete(parser.hidden_names) collect_hidden_names(&parser) skip_newlines(&parser) for current(&parser).kind != .Eof { parse_top_level(&parser) skip_newlines(&parser) } return parser.module } parse_into :: proc( stream: ^token.Stream, source_file: ^source.Source, diagnostics: ^source.Diagnostics, module: ^ast.Module, pkg: ast.Package_Id, file: ast.File_Id, ) { parser := Parser{ tokens=stream, source_file=source_file, diagnostics=diagnostics, module=module^, pkg=pkg, file=file, } parser.hidden_names.allocator = module.allocator defer delete(parser.hidden_names) collect_hidden_names(&parser) skip_newlines(&parser) for current(&parser).kind != .Eof { parse_top_level(&parser) skip_newlines(&parser) } module^ = parser.module }