ebnf | EBNF parser and generic parser generator for Ruby | Parser library

DeclBlock should only be called once in this language. DeclStmt can be called multiple times. A DeclStmt is defined to be one or more Ident followed by :, followed by a type, then ending in a ;.

After you read the ; at the end of a DeclStmt, you'd then read the next token to decide what to do next. If the next token is another Ident you know you're at the start of another DeclStmt, so you'd call that again. If it's anything else, you know you're at the start of ProgBody. (I'm assuming the last bit. Normally you'd look for the token that starts a ProgBody, but that's not shown.)

The | would be the most-loosely bound thing. In other words:

is equivalent to | ([])

and NOT equivalent to ( | )[].

So your first interpretation.

With possible comments (/* ... */) that need be omitted:

I don't think it's helpful, today, to think of closures as some special magic thing: long ago in languages in the prehistory of Scheme they were, but in modern languages they are not a special thing at all: they just follow from the semantics of the language in an obvious way.

The two important things (these are both quotes from R7RS, both from section 1.1 are these:

Scheme is a statically scoped programming language. Each use of a variable is associated with a lexically apparent binding of that variable.

and

All objects created in the course of a Scheme computation, including procedures and continuations, have unlimited extent.

What this means is that Scheme is a language with lexical scope and indefinite extent: any variable binding exists for as long as there is possibility of reference. And, conveniently, you can always tell statically (ie by reading the code) what bindings a bit of code may refer to.

And the important thing here is that these rules are absurdly simple: there are no weird special cases. If a reference to a variable binding is visible in a bit of code, which you can tell by looking at the code, then it is visible. It's not visible only sometimes, or only during some interval, or only if the Moon is gibbous: it's visible.

But the implication of the rules is that procedures, somehow, need to remember all the bindings that they reference or may reference and which were in scope when they were created. Because scope is static it is always possible to determine which bindings are in scope (disclaimer: I'm not sure how this works formally for global bindings).

So then the very old-fashioned definition of a closure would be a procedure defined in a scope in which bindings to which it refers exist. This would be a closure:

The only BNF available is from Appendix G, CLIPS BNF, of the Basic Programming Guide. That covers all of the constructs and function calls that are not special forms. For special forms (functions that have non-standard syntax such as the fact query functions), the BNF is specified in the section describing that function.

It's documented in PEP 617 under Grammar Expressions:

~

Commit to the current alternative, even if it fails to parse.

rule_name: '(' ~ some_rule ')' | some_alt

In this example, if a left parenthesis is parsed, then the other alternative won’t be considered, even if some_rule or ‘)’ fail to be parsed.

The ~ basically indicates that once you reach it, you're locked into the particular rule and cannot move onto the next rule if the parse fails. PEP 617 mentions earlier that | some_alt can be written in the next line.

I don't know if Flex and Bison are the best tools for this particular problem. First, your grammar does not easily fit a model in which the input can be unambiguously divided into tokens, and then those tokens parsed with a limited lookahead context-free grammar.

Perhaps you had in mind that flex would just categorise each incoming symbol and let the parser deal with assembling individual characters into grammatical structures. That can be done, although it's not a common approach. Usually, we would make a word a single token, for example. That will be difficult in this case, though, because there are contexts in which you want a word consisting only of digits to be considered to be a number, and other contexts in which it should be treated as a word (leaving aside for the moment the difficulty of handling text). That could possibly be accomplished, but it would require some tweaking of the grammar, and I'm not at all sure that all ambiguities could be resolved during a left-to-right parse.

It's certainly possible to use Flex only to categorise UTF-8 sequences into individual characters, segregated by general major category (perhaps with some minor refinements). I guess that's what your question is asking about, and the bad news is that Flex really isn't designed for UTF-8 processing, and furthermore Unicode was not designed with the goal of coalescing codepoint ranges. Unicode tries to place codepoints so that characters which come from the same script are all placed in a few aligned blocks of codepoints, leaving some codepoints unused in order to avoid mixing characters from some other script. Moreover, the unused codes are not always at the end of the blocks, because sometimes codepoints are assigned according to some algorithm --like capital letters are at a fixed offset from the equivalent lower-case letter-- which doesn't produce all codepoints (perhaps because sometimes the upper-to-lower case mapping isn't one-to-one.) You can see this at play in the two main blocks of Greek characters, with commonly used characters mostly assigned to the range 0x0370-0x03FF, with accented and other less commonly-used characters in the code block 0x1F00-0x1FFF. Some other blocks are used for historical symbols, so you can spend quite a bit of time exploring that rabbit hole if you're interested enough.)

Once that pattern is repeated over and over again, you find that there are something like a thousand small runs of unassigned characters (category Cn) which interrupt the runs of other character categories. So you could collect all the possible UTF-8 sequences for each character category, but you would end up with a lot of ranges, possibly exceeding Flex's limits on the size of a pattern [See Notes 1 and 2].

Anyway, I think that idea is a dead-end, because you'll have immense problems writing a one-symbol lookahead grammar if the symbols are single characters. It's not even obvious to me how you might write a one-symbol lookahead grammar if you were to tokenise into words. Take, for example, the EBNF productions:

I'm not really sure I understand your question.

If you just have a single production for a non-terminal and that production just has a single repetition operator, either at the beginning or the end, you can apply a simple desugaring: (Here α and β are sequences of grammar symbols (but not EBNF symbols), and α might be empty.)

This document specifies the query syntax: