ebnf | Image generator for EBNF - To see the CI status click

The relevant quote from the "specification" is this:

A squirrel program is a simple sequence of statements.:

stats := stat [';'|'\n'] stats

[...] Statements can be separated with a new line or ‘;’ (or with the keywords case or default if inside a switch/case statement), both symbols are not required if the statement is followed by ‘}’.

These are relatively complex rules and in their totality not context free if newlines can also be ignored everywhere else. Note however that in my understanding the text implies that ; or \n are required when no of the other cases apply. That would make your example illegal. That probably means that the BNF as written is correct, e.g. both ; and \n are optionally everywhere. In that case you can (for lark) just put an %ignore "\n" statement and it should work fine.

Also, lark should not complain if you both ignore the \n and use it in a rule: Where useful it will match it in a rule, otherwise it will just ignore it. Note however that this breaks if you use a Terminal that includes the \n (e.g. WS or /\s/). Just have \n as an extra case.

(For the future: You will probably get faster response for lark questions if you ask over on gitter or at least put a link to SO there.)

The following associativity declarations resolved all shift/reduce conflicts and produced the expected output (in all tests I could think of at least):

Is this right?

No. The grammar show merely says things like “Only switch, break, and continue can have case some-constant : labels, while the others can have some-constant :.

To say that a case label can only be used on commands that are directly inside a switch, the body of a switch, or presumably the body of a compound statement that is the body of a switch, must be a statement or list of statements of a type that can have case labels, whereas the statements or list of statements that appear elsewhere must be statements of a type that cannot have case labels.

So define a grammar token like MayBeLabeledStatement that can have a case label and another grammar token like Statement that cannot have a case label. Both statements may expand to the various kinds of statements (expression, if, while, and so on), but the ones for MayBeLabeledStatement also may have case labels. Further, all the statements within a MayBeLabeledStatement should be Statement type statements, which cannot have case labels—except you may need to think about how to handle the single statement inside a switch (which could be a compound statement or not).

Similarly, you will need different grammar tokens to handle statements inside a while or switch (which allow break) versus those not inside a while or switch and another for those within a while (which allow continue) versus those not inside a while. You may also need some combinations, for statements that are inside a while and are directly inside a switch versus statements that are inside a while and are not directly inside a switch.

Answer is explained better here: https://stackoverflow.com/a/20179940/5556724

The version of EBNF on Wikipedia is written like so:

There's a nice package already available to do that and it's by the Prometheus's Authors itself.

They have written a bunch of Go libraries that are shared across Prometheus components and libraries. They are considered internal to Prometheus but you can use them.

Refer: github.com/prometheus/common doc. There's a package called expfmt that can decode and encode the Prometheus's Exposition Format (Link). Yes, it follows the EBNF syntax so ebnf package could also be used but you're getting expfmt right out of the box.

Package used: expfmt

Sample Input:

The XML standard, as an example, uses the following production for whitespace:

In order to be able to match an arbitrary amount of anything (be it parentheses, operators, list items etc.) you need recursion (EBNF also features repetition operators that can be used instead of recursion in some cases, but not for constructs that need to be matched like parentheses).

For well-matched parentheses, the proper production is simply:

The EBNF production for CharData,

[14] CharData ::= [^<&]* - ([^<&]* ']]>' [^<&]*)

means that XML character data can consist of any characters except

• <, which begins markup (tags, comments, XML declarations, CDATA sections, and PIs)
• &, which begins entity references,
• the string of characters, ]]>, which ends a CDATA section.

Escaping:

• Escape < as < in character data.
• Escape & as & in character data.
• ]]> cannot appear in character data; there is no escaped form.

See also:

• Minus in w3c specification grammar

If you are interested in speed, and you do not want to worry for stack overflow occurrences, you may try Tunnel Grammar Studio, a parser generator I work on. The generated parsers from it are iterative during lexing, parsing, the tree construction, tree iteration, tree to string conversion and tree release. The accepted grammars are in ABNF (RFC 5234) with case sensitivity per token (RFC 7405).

Its a good idea to have a deterministic grammar with any parser you use. TGS does check for LL(1) conflicts at compile time, and will help you to create a deterministic grammar by visualizing the conflict places.

There is a demo of the tool, you can test the speed yourself. There is an option in the tool to generate fully ready test case project that will log into the console at runtime the time taken to parse, iterate the tree and free it, by only supplying the input data. Meaning that no development (except compiling the generated code) from you is expected if you want to test the speed for a grammar.

In my tests with a JSON grammar (RFC 8259) with removed ambiguity, that only emits syntax tree build events (like SAX) the iterative parser runs with around 8 megabytes per second, that are many lines per second and takes memory only proportional to the depth of the parsing, because only one token is technically needed at runtime for LL(1) grammars, i.e. it is practically "streaming" the input.

You can have also statically typed or dynamically typed concrete syntax tree, or dynamically typed abstract syntax tree with different levels of abstraction (i.e. automatic node pruning). The syntax trees builders (if chosen) for this trees are using the build events to create the relevant trees. You will need an ABNF grammar and C++ as a language target however.

The tool supports token ranges inside the parser grammar (additionally to the character ranges inside the lexer grammar). This means that you may develop your grammar without the extra care of the lexical rules order.