normalizer | Input Normalization Package

You need to use script_score along with the function score query.

Try out this below query

I think what you want to do is to merge your decoded entity to be able to use Doctrine abilities ?

📖 Take a look on this documentation of Doctrine : https://www.doctrine-project.org/projects/doctrine-orm/en/2.8/reference/working-with-objects.html#merging-entities

It would be something like :

You need to use script_score as defined in the search query below along with the function score query.

Adding a working example with index data, mapping, search query and search result

Index Mapping:

Your approach has an overall problem: the KNeighborsClassifier does not have an intrinsic measure of feature importance. Thus, it is not compatible with RFECV as its documentation states about the classifier:

A supervised learning estimator with a fit method that provides information about feature importance either through a coef_ attribute or through a feature_importances_ attribute.

You will definitely fail with KNeighborsClassifier. You definitely need another classifier like RandomForestClassifier or SVC.

If you can shoose another classifier, your pipeline still needs to expose the feature importance of the estimator in your pipeline. For this you can refer to this answer here which defines a custom pipeline for this purpose:

According to the documentation, LDA includes a featuresCol argument, with default value featuresCol='features', i.e. the name of the column that holds the actual features; according to your shown schema, such a column is not present in your dataframe, hence the expected error.

It is not exactly clear which column contains the features in your dataframe - get_features or get_idf_feature (they look identical in the sample you show); assuming it is get_idf_feature, you should change the LDA call to:

So I figured this out myself, finally! There is some error in huggingface code so i loaded the tokenizer like this and it worked.

You are using Hibernate Search 5, which is old, with a recent version of Spring.

Hibernate Search 5, being old, depends on an old version of the Elasticsearch client JARs.

Your version of Spring depends on a newer, incompatible version of the Elasticsearch client JARs.

Two solutions:

1. Upgrade to the newer Hibernate Search 6, which will most likely depend on a version of Elasticsearch that is compatible with the one bundled with Spring. Note that just changing the version won't be enough: group IDs, artifact IDs and APIs are different. See the migration guide.

2. OR try to downgrade to an older version of the Elasticsearch client. For example, with Spring Boot:

You are getting a mapper parsing exception, which states that

"analyzer on field [keyword_analyzered_field] must be set when search_analyzer is set"

"search_analyzer" can be included in the index mapping definition of a field, only if the "analyzer" is defined for that particular field.

Modify your index mapping as

In the second prod_pdf you are using computed PDFs while in the first you were using defined distributions. So, in the second prod_pdf you already have the PDF. Thus, in the for loop you simply need to do p_pdf = p_pdf * pdf

From the paper you linked, we know that "For discrete input distributions, the analogous definition of conflation is the normalized product of the probability mass functions". So you need not only to take the product of PDFs but also to normalize it. Thus, rewriting the equation for a discrete distribution, we get

where F is the number of distributions we need to conflate and N is the length of the discrete variable x.