punctuator | LSTM RNN for restoring missing punctuation | Machine Learning library

as far as I am concerned, C does use cdecl

Its name notwithstanding, the cdecl convention is not universal for C code, not even on the x86 architecture. It has the advantage of being simple to define and implement, but it makes no use of CPU registers for argument passing, which is more efficient. That makes a difference even on register-starved x86, but it makes a lot more difference on architectures with more available registers, such as x86_64.

Talking about the cleanup, here is my question. I do not understand: are there three different things?

1. stack clean
2. moving the pointer back to the penultimate stack frame
3. stack restoration

Or how should I see them?

I would be inclined to interpret (1) and (3) as different ways of saying the same thing, but it is conceivable that someone would draw distinctions between them. (3) and related wording is what I encounter most frequently. (2) is not necessarily the same thing, because there may be two relevant stack parameters to be restored: the base of the stack frame (see below), and the top of the stack. The stack frame base is important in the event that the stack frame contains more information than argument and local variable values, such as the base of the previous stack frame.

Also, the target of this question is basically how could variadic function works in calling conventions like Pascal or stdcall where the callee should clear / clean / restore (I don't know which operation) the stack - but he doesn't know how many parameters it will receive.

The stack is not necessarily the whole picture.

The callee cannot restore the stack if it does not know how to find the top of its caller's stack, and, if necessary, the base of its caller's stack frame. But in practice, this is usually hardware assisted.

Taking x86 (for which cdecl was designed) as an example, the CPU has registers for both the stack (frame) base and the current stack pointer. The caller's stack base is stored on the stack at a known offset (0) from the callee's stack base. Regardless of the number of arguments, the callee restores the stack by moving the top of the stack to its own stack base, and popping the value there to obtain the caller's stack base.

It is conceivable, however, that there is a calling convention in use somewhere that affords no way to restore the stack to a chosen previous state other than to pop elements one at a time, that does not explicitly convey the number of arguments to the called function, and that requires the callee to restore the caller's stack. Such a calling convention would not support variadic functions.

Why is it so important the order in which parameters are pushed on to the stack?

The order is not important in any general sense, but it is essential for caller and callee, which may be compiled separately, to agree about it. Otherwise, the callee cannot match the passed values with the parameters they are intended for. Thus, to whatever extent a calling convention relies on the stack, it must specify precisely which arguments are passed there, and in which order.

Regarding stack frames: this is more material that is not specified by C and that varies, at least to some extent. Conceptually, though, the stack frame of a function call is the portion of the stack that provides execution context for that call. It typically supplies storage for local variables, and it may contain additional information, such as a return address and / or the value of the caller's stack frame pointer. It might also contain other per-function-call information appropriate for the execution environment. Details are part of the calling convention in use.

The complete lexical specification for ECMAScript (aka JavaScript) can be found in Section 11 (Lexical Grammar) of ECMA 262. (New versions of ECMA-262 are released roughly annually, but the URL in that link -- correct as of ECMAScript 2020 -- should continue to work, or be easy to fix. I didn't transcribe the list of reserved words, semi-reserved words, operators and other punctuators, because those lists may well change in a future standard.)

However, precisely specifying which IdentifierNames have specific meaning is not simple. You need to carefully read subsection 11.6 (Names and Keywords), and it will probably still be a bit confusing. The difficulty is that each successive version of ECMAScript tries extraordinarily hard to stay compatible with previous versions, making it hard to introduce new keywords. So many significant symbols are not in the list of reserved words. Some, like await are only reserved in certain contexts; many others (such as let) are only reserved in strict mode; and a few (such as async) are always usable as identifiers, but can be used syntactically in some contexts where an identifier would be a syntax error. There are various lists of symbols in section 11.6.2, which you could combine or select from, depending on what your need is.

Operator symbols (other than word operators) are a bit more straightforward (although there are still some oddities, so reading the fine print is still necessary). See the various lists in section 11.7.

It is possible to mix and match the Processor API with the DSL, but you can't process a KTable. You would need to convert to a KStream. Alternatively you could create a new topology with a Processor that interacts with the state store.

You will need to store that state somewhere - how to determine if records are older than one hour. One option could be to add a timestamp to each record in the state store.

In the init method of a Processor you could call schedule (punctuate) to iterate records in the state store and remove old ones:

I think one thing you could do is the following:

The C++ standard document actually appears to not define what punctuator means, other than punctuators being a lexical token. Specifically:

[gram.lex]

RocksDB does file compaction in the background. Hence, if you need a more aggressive compaction you should pass in a custom RocksDBConfigSetter via Streams config parameter rocksdb.config.setter. For more details about RockDB, check out the RocksDB documentation.

https://docs.confluent.io/current/streams/developer-guide/config-streams.html#rocksdb-config-setter

However, I would not recommend to change RocksDB configs as long as there is no real issue -- you can do more harm than good. Seems you store size is quite small, thus, I don't see a real problem atm.

Btw: You should chance state.dir config if you go to production and not put state into the default /tmp location.

If you pay attention, you'll remark that the tokens are described with a regular grammar, this means that they could also be described with regular expressions. Why the editor of the standard preferred one formalism to the other is open to speculation, you could think that using only one formalism for both part was considered simpler.

The rules for white space and comments hint that the separation of concern between the parser and the lexer was present at the mind of the designer. You can't use the description as is in an lexer-less parser.

Note that the preprocessor is reason for the introduction of preprocessing-token. Things like header-name and pp-number have consequence on the behavior of the preprocessor. Note also that some tokens are recognized only in some contexts (notably

Registering a punctuation will not spawn a new thread. The number of used threads in determined by num.stream.threads configuration only. Hence, if you register a punctuation, it's executed on the same thread as the topology and thus it is thread safe.

If you configure more threads than available input topic partitions, some threads would not get any work assigned, and thus, they would not execute any punctuations.

As usual, I found the answer in the javadoc about ValueTransformerWithKey interface: https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/kstream/ValueTransformerWithKey.html

Note, that using ProcessorContext.forward(Object, Object) or ProcessorContext.forward(Object, Object, To) is not allowed within transform and will result in an exception.

However, implementing the Transformer interface instead allows the usage of context.forward(). Thanks @Matthias J. Sax

https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/kstream/Transformer.html

If more than one output record should be forwarded downstream ProcessorContext.forward(Object, Object) and ProcessorContext.forward(Object, Object, To) can be used. If record should not be forwarded downstream, transform can return null.