OpenSearch | 🔎 Open source distributed and RESTful search engine | Search Engine library

You have a named export for MovieProvider and in the same file a default export for MovieContext;

Each VM has some overhead for the OS so 10 smaller instances would have less compute and RAM available for ES in total than 5 larger instances. Also, if you just leave the default index settings (5 primary shards, 1 replica) and actively write to only 1 index at a time, you'll effectively have only 5 nodes indexing data for you (and these nodes will have less bandwidth because they are smaller).

So, I would usually recommend running a few larger instances instead of many smaller ones. There are some special cases where it won't be true (like a concurrent-search-heavy cluster) but for those, I'd recommend going with even larger instances in the first place.

You need to replace page by _doc (or remove it altogether) as there's no more mapping types

The AWS::OpenSearchService::Domain CloudFormation resource has a property called AdvancedOptions.

As per documentation, you should pass override_main_response_version to the advanced options to enable compatibility mode.

Example:

Firehose uses lambda function to transform records before they are being delivered to the destination in your case OpenSearch(ES) so they are only used to modify the structure of the data but can't be used to influence CRUD actions. Firehose can only insert records into a specific index. If you need a simple option to remove records from ES index after a certain period of time have a look at "Index rotation" option when specifying destination for your Firehose stream.

If you want to use CRUD actions with ES and keep using Firehose I would suggest to send records to S3 bucket in the raw format and then trigger a lambda function on object upload event that will perform a CRUD action depending on fields in your payload.

A good example of performing CRUD actions against ES from lambda https://github.com/chankh/ddb-elasticsearch/blob/master/src/lambda_function.py

This particular example is built to send data from DynamoDB streams into ES but it should be a good starting point for you

You can define null_value properties while configuring index mapping.

have you seen this article from official side? Pay attention on Log files overview section.

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to. C:\k\kubelet.err.log

Also you can find Fluent GitHub Community and create an issue there to have better support from its contributors

There is a Slack channel for Fluent

Q : " Writing to a file parallely while processing in a loop in python ... "

A :
Frankly speaking, the file-I/O is not your performance-related enemy.

_{"With all due respect to the colleagues, Python (since ever) used GIL-lock to avoid any level of concurrent execution ( actually re-SERIAL-ising the code-execution flow into dancing among any amount of threads, lending about 100 [ms] of code-interpretation time to one-AFTER-another-AFTER-another, thus only increasing the interpreter's overhead times ( and devastating all pre-fetches into CPU-core caches on each turn ... paying the full mem-I/O costs on each next re-fetch(es) ). So threading is ANTI-pattern in python (except, I may accept, for network-(long)-transport latency masking ) – user3666197 44 mins ago "}

Given about the 65k files, listed in CSV, ought get processed ASAP, the performance-tuned orchestration is the goal, file-I/O being just a negligible ( and by-design well latency-maskable ) part thereof _{( which does not mean, we can't screw it even more ( if trying to organise it in another performance-devastating ANTI-pattern ), can we? )}

Tip #1 : avoid & resist to use any low-hanging fruit SLOCs if The Performance is the goal

If the code starts with a cheapest-ever iterator-clause,
be it a mock-up for aRow in aCsvDataSET: ...
or the real-code for i in range( len( queries ) ): ... - these (besides being known for ages to be awfully slow part of the python code-interpretation capabilites, the second one being even an iterator-on-range()-iterator in Py3 and even a silent RAM-killer in Py2 ecosystem for any larger sized ranges) look nice in "structured-programming" evangelisation, as they form a syntax-compliant separation of a deeper-level part of the code, yet it does so at an awfully high costs impacts due to repetitively paid overhead-costs accumulation. A finally injected need to "coordinate" unordered concurrent file-I/O operations, not necessary in principle at all, if done smart, are one such example of adverse performance impacts if such a trivial SLOC's ( and similarly poor design decisions' ) are being used.

Better way?

• a ) avoid the top-level (slow & overhead-expensive) looping
• b ) "split" the 65k-parameter space into not much more blocks than how many memory-I/O-channels are present on your physical device ( the scoring process, I can guess from the posted text, is memory-I/O intensive, as some model has to go through all the texts for scoring to happen )
• c ) spawn n_jobs-many process workers, that will joblib.Parallel( n_jobs = ... )( delayed( <_scoring_fun_> )( block_start, block_end, ...<_params_>... ) ) and run the scoring_fun(...) for such distributed block-part of the 65k-long parameter space.
• d ) having computed the scores and related outputs, each worker-process can and shall file-I/O its own results in its private, exclusively owned, conflicts-prevented output file
• e ) having finished all partial block-parts' processing, the main-Python process can just join the already ( just-[CONCURRENTLY] created, smoothly & non-blocking-ly O/S-buffered / interleaved-flow, real-hardware-deposited ) stored outputs, if such a need is ...,
  and
  finito - we are done ( knowing there is no faster way to compute the same block-of-tasks, that are principally embarrasingly independent, besides the need to orchestrate them collision-free with minimised-add-on-costs).

If interested in tweaking a real-system End-to-End processing-performance,
start with lstopo-map
next verify the number of physical memory-I/O-channels
and
may a bit experiment with Python joblib.Parallel()-process instantiation, under-subscribing or over-subscribing the n_jobs a bit lower or a bit above the number of physical memory-I/O-channels. If the actual processing has some, hidden to us, maskable latencies, there might be chances to spawn more n_jobs-workers, until the End-to-End processing performance keeps steadily growing, until a system-noise hides any such further performance-tweaking effects

A Bonus part - why un-managed sources of latency kill The Performance

As mentioned in the comments content of website is provided dynamically, so you won't get your information with requests on that specific ressource / url, cause it is not able to render the website like a browser would do.

It do not need beautifulsoup for that task, cause there are ressources that will give you structured json data: