preprocess | Corpus preprocessing | Natural Language Processing library

I tried with scikeras but I got errors because it doesn't accept not-numerical inputs (in our case the input is in str format). So I came back to the standard keras wrapper.

The focal point here is that the model is not built correctly. The TextVectorization must be put inside the Sequential model like shown in the official documentation.

So the build_model function becomes:

This is a bug in GCC. C 2018 6.10.3.2 specifies behavior of the # operator. Paragraph 1 says “Each # preprocessing token in the replacement list for a function-like macro shall be followed by a parameter as the next preprocessing token in the replacement list.” We see this in the #x of #define STR_(x) #x.

Paragraph 2 says:

If, in the replacement list, a parameter is immediately preceded by a # preprocessing token, both are replaced by a single character string literal preprocessing token that contains the spelling of the preprocessing token sequence for the corresponding argument. Each occurrence of white space between the argument’s preprocessing tokens becomes a single space character in the character string literal. White space before the first preprocessing token and after the last preprocessing token composing the argument is deleted…

The X(Y,Y) macro invocation must have resulted in the tokens Y and Y, and we see in #define X(x,y) x y that they would have white space between them.

White-space in a macro replacement list is significant, per 6.10.3 1, which says:

Two replacement lists are identical if and only if the preprocessing tokens in both have the same number, ordering, spelling, and white-space separation, where all white-space separations are considered identical.

Thus, in #define X(x,y) x y, the replacement list should not be considered to be just the two tokens x and y, with white space disregarded. The replacement list is x, white space, and y.

Further, when the macro is replaced, it is replaced by the replacement list (and hence includes white space), not merely by the tokens in the replacement list, per 6.10.3 10:

… Each subsequent instance of the function-like macro name followed by a ( as the next preprocessing token introduces the sequence of preprocessing tokens that is replaced by the replacement list in the definition (an invocation of the macro)… Within the sequence of preprocessing tokens making up an invocation of a function-like macro, new-line is considered a normal white-space character.

You need to look at 6.10.1p1 in its entirety:

The expression that controls conditional inclusion shall be an integer constant expression except that: identifiers (including those lexically identical to keywords) are interpreted as described below;¹⁶⁶⁾, and it may contain unary operator expressions of the form

You can use a 2D FFT to find the general direction in which the crop is aligned (as proposed by mozway in the comments). The idea is that the general direction can be easily extracted from centred beaming rays appearing in the magnitude spectrum when the input contains many lines in the same direction. You can find more information about how it works in this previous post. It works directly with the input image, but it is better to apply the Gaussian + Canny filters.

Here is the interesting part of the magnitude spectrum of the filtered gray image:

The main beaming ray can be easily seen. You can extract its angle by iterating over many lines with an increasing angle and sum the magnitude values on each line as in the following figure:

Here is the magnitude sum of each line plotted against the angle (in radian) of the line:

Based on that, you just need to find the angle that maximize the computed sum.

Here is the resulting code:

You probably have something like this.

I found a partial answer. Both StandardScaler and SelectKBest have .get_feature_names_out methods. I did not find the time to investigate further.

using OneHotEncoder is not the way to go here, it's better to extract the features from the column time as separate features like year, month, day, hour, minutes etc... and give these columns as input to your model.

Generally, you can try something like this:

This isn't supported in SKLearnProcessor. You'd need to package your dependencies in docker image and create a custom Processor (e.g. a ScriptProcessor with the image_uri of the docker image you created.)