Ivory | a language based on Diksam | Interpreter library

As noted in comments, identifiers in SML (as in many other programming languages) cannot begin with digits.

First off, just to rain a little on your parade I just want to point out that Wilfred already made a brainf*ck compiler in Rust that can compile to a native binary through LLVM (bfc). If you are getting stuck, you may want to check his implementation to see how he does it. If you ignore the LLVM part, it isn't too difficult to read through and he has a good approach.

When broken down into its core components, this problem revolves around merging two elements together. The most elegant way I have come across to solve this is use an iterator with a merge function. I wrote an example of what I imagine that would look like below. I shortened some of the variable names since they were a bit long but the general idea is the same. The merge function has a very simple job. When given two elements, attempt to merge them into a single new element. The iterator then handles putting them through that function and returning items once they can no longer be merged. A sort of optional fold if you will.

Using the command prompt to code is different than using IDE. When you use the command prompt you use something called interpreter, it will execute every line you write right after you click enter. You can write an object like that:

Go to File | New Projects Setup | Preferences for New Projects (Settings for New Projects on Windows). Select Python Interpreter settings.

If you want to guarantee termination you can conservatively assume any input goal is nonterminating until proven otherwise, using a decidable proof procedure. Basically, define some small class of goals which you know halt, and expand it over time with clever ideas.

Here are three examples, which guarantee or force three different kinds of termination respectively (also see the Power of Prolog chapter on termination):

• existential-existential: at least one answer is reached before potentially diverging
• universal-existential: no branches diverge but there may be an infinite number of them, so the goal may not be universally terminating
• universal-universal: after a finite number of steps, every answer will be reached, so in particular there must be a finite number of answers

In the following, halts(Goal) is assumed to correctly test a goal for existential-existential termination.

This uses halts/1 to prove existential termination of a modest class of goals. The current evaluator eval/1 just falls back to the underlying engine:

Using VBCodeProvider you can compile VB.NET code at run-time.

The following example, compiles a piece of VB.NET code at run-time and run it:

The files have been copied from the base Python installation and carries the modification time from there.

The files' modification times shouldn't cause any "problems in selecting interpreters".

I think you'll agree that most languages (likely including the one you are implementing) have conceptual tokens:

• operators, e.g * (usually multiply), '(', ')', ;
• keywords, e.g., "IF", "GOTO"
• identifiers, e.g. FOO, count, ...
• numbers, e.g. 0, -527.23E-41
• comments, e.g., /* this text is ignored in your file */
• whitespace, e.g., sequences of blanks, tabs and newlines, that are ignored

As a practical matter, it takes a specific chunk of code to scan for/collect the characters that make each individual token. You'll need such a code chunk for each type of token your language has.

If you write a parser without a lexer, at each point where your parser is trying to decide what comes next, you'll have to have ALL the code that recognize the tokens that might occur at that point in the parse. At the next parser point, you'll need all the code to recognize the tokens that are possible there. This gives you an immense amount of code duplication; how many times do you want the code for blanks to occur in your parser?

If you think that's not a good way, the obvious cure to is remove all the duplication: place the code for each token in a subroutine for that token, and at each parser place, call the subroutines for the tokens. At this point, in some sense, you already have a lexer: an isolated collection of code to recognize tokens. You can code perfectly fine recursive descent parsers this way.

The next thing you'll discover is that you call the token subroutines for many of the tokens at each parser point. Even that seems like a lot of work and duplication. So, replace all the calls with a single "GetNextToken" call, that itself invokes the token recognizing code for all tokens, and returns a enum that identifies the specific token encountered. Now your parser starts to look reasonable: at each parser point, it makes one call on GetNextToken, and then branches on enum returned. This is basically the interface that people have standardized on as a "lexer".

One thing you will discover is the token-lexers sometimes have trouble with overlaps; keywords and identifiers usually have this trouble. It is actually easier to merge all the token recognizers into a single finite state machine, which can then distinguish the tokens more easily. This also turns out to be spectacularly fast when processing the programming language source text. Your toy language may never parse more than 100 lines, but real compilers process millions of lines of code a day, and most of that time is spent doing token recognition ("lexing") esp. white space suppression.

You can code this state machine by hand. This isn't hard, but it is rather tedious. Or, you can use a tool like FLEX to do it for you, that's just a matter of convenience. As the number of different kinds of tokens in your language grows, the FLEX solution gets more and more attractive.

TLDR: Your parser is easier to write, and less bulky, if you use a lexer. In addition, if you compile the individual lexemes into a state machine (by hand or using a "lexer generator"), it will run faster and that's important.

You are going to have to change your logic to not use recursion for reading each line. If you use recursion you will add a new function call to the stack for every layer of recursion, and the prior function calls will not be removed until the exit condition is met. If you make it 500 or so calls deep you can expect a stackoverflow exception.

Now, I don't have time to read over your code, but I can tell you what you need to do: Turn your recursive call into a loop.

Your code can probably be broken down into something like this: