Luigi | 類似文書をお手軽＆高速に検索 - This is Perl module Luigi

I have a test pipeline on concourse with one job that runs a set of luigi tasks. My problem is: failures in the luigi tasks do not rise up to the concourse job. In other words, if a luigi task fails, concourse will not register that failure and states that the concourse job completed successfully. I will first post the code I am running, then the solutions I have tried.

luigi-tasks.py

the IP range written in this way:

10.27.0.0/16

means that the address

10.27.24.152

is included?

Thanks a lot.

Luigi

I'm using nat bib for beamer and my citations are all showing up with n.d., despite the citations having a date in the bib file.

I have an index with 1 million phrases and I want to search in the index with some query phrases in italian (and that is not the problem). The problem is in the order in which the matches are retrieved: I want to have first the exact matches so I changed the default similarity to "boolean" and I thought it was a good idea but sometimes it does not work. For example: searching in my index for phrases containing the words "film cortometraggio" the first matches are:

• Distribuito dalla General Film Company, il film- un cortometraggio in due bobine
• Distribuito dalla General Film Company, il film - un cortometraggio di 150 metri - uscì nelle sale cinematografiche

But there are some better phrases that should be returned before those ones like:

• Robinet aviatore Robinet aviatore è un film cortometraggio del 1911 diretto da Luigi Maggi;

This last phrase should be returned first in my opinion because there is no space between the two words I am searching for.

Using the BM25 algorithm the first match that I get is "Pappi Corsicato Ha diretto film, cortometraggi, documentari e videoclip.". In this case also should be provided the phrase "Robinet aviatore Robinet aviatore è un film cortometraggio del 1911 diretto da Luigi Maggi;" because is an exact match and I don't get why the algorithm gives the other phrase a higher score.

I am using the Java Rest high level client and the search query that I'm doing are simple match Phrase query, like this: searchSourceBuilder.query(QueryBuilders.matchPhraseQuery(field, text).slop(5)

This is the structure of the documents in my index:

I have a couple of methods here that are taking longer than I would like to. I'm currently hitting a wall since I don't see any obvious way to write these methods in a more efficient way.

For background, what the code is doing is processing a sales dataset, in order to find previous sales orders related to the same client. However, as you will see, there's a lot of business logic in the middle which is probably slowing things down.

I was thinking about refactoring this into a PySpark job but before I do so, I would like to know if that's even the best way to get this done.

I will highly appreciate any suggestions here.

More context: Each loop is taking about 10 minutes to complete. There are about 24k rows in search_keys. These methods are part of a Luigi task.

I'm trying to create a nested dictionary that tells me what document each word appears in and in which position it appears in: For example:

I wanted to create a variable for each grep regex line from Usernames.txt file. The text contains this

I have difficulty to iterate hashmap and return the keyset with the maximum integer into an HashMap...I leave an example can anyone explain me how to do, thanks.

I'm pretty sure there's a better way to do this: