Documentation
¶
Overview ¶
Package token defines 2 types of Token, constant ones (with no "value") and the ones with attached value that is variable (e.g. IDENT, INT, FLOAT, STRING, *COMMENT). We might have used the upcoming unique https://tip.golang.org/doc/go1.23#new-unique-package but we want this to run on 1.22 and earlier and rolled our own, not multi threaded.
Index ¶
Constants ¶
This section is empty.
Variables ¶
Functions ¶
func ResetInterning ¶ added in v0.25.0
func ResetInterning()
Types ¶
type GrolInfo ¶ added in v0.35.0
type GrolInfo struct {
// Keywords is a map of all known keywords.
Keywords sets.Set[string]
Builtins sets.Set[string]
Tokens sets.Set[string]
}
GrolInfo enables introspection of known keywords and tokens and operators.
type Token ¶
type Token struct {
// contains filtered or unexported fields
}
func ByType ¶ added in v0.25.0
ByType is the cheapest lookup for all the tokens whose type only have one possible instance/value (ie all the tokens except for the first 4 value tokens). TODO: codegen all the token constants to avoid needing this function. (even though that's better than string comparisons).
func ConstantTokenChar ¶ added in v0.25.0
func ConstantTokenChar2 ¶ added in v0.27.0
func InternToken ¶ added in v0.25.0
InternToken looks up a unique pointer to a token of same values, if it exists, otherwise store the passed in one for future lookups.
func LookupIdent ¶
func (*Token) DebugString ¶ added in v0.25.0
type Type ¶
type Type uint8
const ( ILLEGAL Type = iota EOL IDENT // add, foobar, x, y, ... INT // 1343456 FLOAT // .5, 3.14159,... STRING // "foo bar" or `foo bar` LINECOMMENT BLOCKCOMMENT REGISTER // not used for parsing, only to tag object.Register as ast node of unique type. ASSIGN PLUS // order must be PLUS,MINUS,ASTERISK,SLASH,BITAND,BITOR,BITXOR MINUS // additionally, PLUS - SUMASSIGN must = MINUS - SUBASSIGN = ASTERISK - PRODASSIGN = SLASH - DIVASSIGN ASTERISK SLASH BITAND BITOR BITXOR BITNOT BANG PERCENT LT GT COMMA SEMICOLON LPAREN RPAREN LBRACE RBRACE LBRACKET RBRACKET COLON DOT LTEQ GTEQ EQ NOTEQ INCR DECR DOTDOT OR AND LEFTSHIFT RIGHTSHIFT LAMBDA // => lambda short cut: `a,b => a+b` alias for `func(a,b) {a+b}` DEFINE // := (force create new variable instead of possible ref to upper scope) SUMASSIGN // order must be sumassign,subassign,prodassign,divassign,andassign,orassign,xorassign SUBASSIGN PRODASSIGN DIVASSIGN ANDASSIGN ORASSIGN XORASSIGN FUNC TRUE FALSE IF ELSE RETURN FOR BREAK CONTINUE MACRO QUOTE UNQUOTE LEN FIRST REST PRINT PRINTLN LOG ERROR CATCH DEL EOF )
Source Files