BM25 | A Python implementation of the BM25 ranking function | Search Engine library

Let's say there is an existing index with a customized BM25 similarity metric like this:

I want to implement Elasticsearch on a customized corpus. I have installed elasticsearch of version 7.5.1 and I do all my work in python using the official client.

Here I have a few questions:

1. How to customize preprocess pipeline? For example, I want to use a BertTokenizer to convert strings to tokens instead of ngrams
2. How to customize scoring function of each document w.r.t. the query? For example, I want to compare effects of tf-idf with bm25, or even using some neural models for scoring.

If there is great tutorial in python, please share with me. Thanks in advance.

I have an index with 1 million phrases and I want to search in the index with some query phrases in italian (and that is not the problem). The problem is in the order in which the matches are retrieved: I want to have first the exact matches so I changed the default similarity to "boolean" and I thought it was a good idea but sometimes it does not work. For example: searching in my index for phrases containing the words "film cortometraggio" the first matches are:

• Distribuito dalla General Film Company, il film- un cortometraggio in due bobine
• Distribuito dalla General Film Company, il film - un cortometraggio di 150 metri - uscì nelle sale cinematografiche

But there are some better phrases that should be returned before those ones like:

• Robinet aviatore Robinet aviatore è un film cortometraggio del 1911 diretto da Luigi Maggi;

This last phrase should be returned first in my opinion because there is no space between the two words I am searching for.

Using the BM25 algorithm the first match that I get is "Pappi Corsicato Ha diretto film, cortometraggi, documentari e videoclip.". In this case also should be provided the phrase "Robinet aviatore Robinet aviatore è un film cortometraggio del 1911 diretto da Luigi Maggi;" because is an exact match and I don't get why the algorithm gives the other phrase a higher score.

I am using the Java Rest high level client and the search query that I'm doing are simple match Phrase query, like this: searchSourceBuilder.query(QueryBuilders.matchPhraseQuery(field, text).slop(5)

This is the structure of the documents in my index:

I am deploying a simple text retrieval system with Vespa. However, I found when setting topk to some large number, e.g. 40, the response will include the error message "Summary data is incomplete: Timed out waiting for summary data." and also some unexpected ids. The system works fine for some small topk like 10. The response was as follows:

{'root': {'id': 'toplevel', 'relevance': 1.0, 'fields': {'totalCount': 1983140}, 'coverage': {'coverage': 19, 'documents': 4053984, 'degraded': {'match-phase': False, 'timeout': True, 'adaptive-timeout': False, 'non-ideal-state': False}, 'full': False, 'nodes': 1, 'results': 1, 'resultsFull': 0}, 'errors': [{'code': 12, 'summary': 'Timed out', 'message': 'Summary data is incomplete: Timed out waiting for summary data. 1 responses outstanding.'}], 'children': [{'id': 'index:square_datastore_content/0/34b46b2e96fc0aa18ed4941b', 'relevance': 44.44359956427316, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/16dbc34c5e77684cd6f554fd', 'relevance': 43.94371735208669, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/9f2fd93f6d74e88f96d7014f', 'relevance': 43.298002713993384, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/76c4e3ee15dc684a78938a9d', 'relevance': 40.908658368905485, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/c04ceee4b9085a4d041d8c81', 'relevance': 36.13561898237115, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/13806c518392ae7b80ab4e4c', 'relevance': 35.688377118163714, 'source': 'square_datastore_content'}, {'id': 'index:square_datastore_content/0/87e0f13fdef1a1c404d3c8c6', 'relevance': 34.74150232183567, 'source': 'square_datastore_content'}, ...]}}

I am using the schema:

I was trying to follow the tutorial - http://ethen8181.github.io/machine-learning/search/bm25_intro.html#ElasticSearch-BM25

I successfully started my elastic node by running as a daemon and it did respond upon issuing the query - curl -X GET "localhost:9200/

When I try running the following code here, it returns 400.

I know when we use the filter function, we could apply a LOWER()/UPPER() function to match our search criterion.

I have a sample Vespa instance and I want to train a lightgbm model from the rank-profile. https://docs.vespa.ai/documentation/learning-to-rank.html

However, anytime I specify the recall with the docID, I get 0 hits. I'm using example code from here: https://github.com/vespa-engine/sample-apps/blob/master/text-search/src/python/collect_training_data.py

how can I improve recall for this condition ?any suggestion? I want to create an index with 39 million passages each one containing at least four sentences in English. My queries are short and interrogative sentences. I know that a language model with Dirichlet smoothing, stop word removal and stemmer is best for this condition. how can I index with these conditions (I've indexed with this configs but there is no difference in results with default bm25)

My index:

I am developing some search engine type application. My Code is look like this: