elasticsearch | Free and Open , Distributed , RESTful Search Engine | Search Engine library

by elastic Java Version: 7.17.8 License: Non-SPDX

kandi X-RAY | elasticsearch Summary

elasticsearch is a Java library typically used in Database, Search Engine, Docker, Spark applications. elasticsearch has build file available and it has high support. However elasticsearch has 1803 bugs, it has 21 vulnerabilities and it has a Non-SPDX License. You can download it from GitHub, Maven.

Elasticsearch is the distributed, RESTful search and analytics engine at the heart of the Elastic Stack. You can use Elasticsearch to store, search, and manage data for:. To learn more about Elasticsearch’s features and capabilities, see our product page.

Support

Quality

Security

License

Reuse

kandi-support Support

elasticsearch has a highly active ecosystem.
It has 62489 star(s) with 22650 fork(s). There are 2708 watchers for this library.
There were 10 major release(s) in the last 6 months.
There are 3543 open issues and 27904 have been closed. On average issues are closed in 358 days. There are 471 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of elasticsearch is 7.17.8

elasticsearch Support

Best in #Search Engine

Average in #Search Engine

elasticsearch Support

Best in #Search Engine

Average in #Search Engine

quality kandi Quality

elasticsearch has 1803 bugs (72 blocker, 50 critical, 880 major, 801 minor) and 66385 code smells.

elasticsearch Quality

Best in #Search Engine

Average in #Search Engine

elasticsearch Quality

Best in #Search Engine

Average in #Search Engine

security Security

elasticsearch has 4 vulnerability issues reported (0 critical, 0 high, 3 medium, 1 low).
elasticsearch code analysis shows 17 unresolved vulnerabilities (5 blocker, 7 critical, 4 major, 1 minor).
There are 693 security hotspots that need review.

elasticsearch Security

Best in #Search Engine

Average in #Search Engine

elasticsearch Security

Best in #Search Engine

Average in #Search Engine

license License

elasticsearch has a Non-SPDX License.
Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

elasticsearch License

Best in #Search Engine

Average in #Search Engine

elasticsearch License

Best in #Search Engine

Average in #Search Engine

build Reuse

elasticsearch releases are available to install and integrate.
Deployable package is available in Maven.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.

elasticsearch Reuse

Best in #Search Engine

Average in #Search Engine

elasticsearch Reuse

Best in #Search Engine

Average in #Search Engine

Top functions reviewed by kandi - BETA

kandi has reviewed elasticsearch and discovered the below as its top functions. This is intended to give you an instant insight into elasticsearch implemented functionality, and help decide if they suit your requirements.

Matches a statement

Initialize the reserved roles .

Gets the main index mappings .

Gets the legal cast .

Loads a whitelist from a resource file .

Scale the cluster to the master node .

Process a new cluster info .

Build table with header information .

Matches an R statement .

Seek to an absolute frame .

Get all kandi verified functions for this library.

elasticsearch Key Features

Logs

Metrics

A search backend

Application monitoring

Endpoint security

elasticsearch Examples and Code Snippets

See all related Code Snippets

Community Discussions

Trending Discussions on elasticsearch

Why should I run multiple elasticsearch nodes on a single docker host?
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead
EFK system is build on docker but fluentd can't start up
How to specify schema for nested json in flask_restx?
How to create a company specific parent dependency in gradle
ElasticSearch - calling UpdateByQuery and Update in parallel causes 409 conflicts
Fuzzy Matching in Elasticsearch gives different results in two different versions
Using Docker-Desktop for Windows, how can sysctl parameters be configured to permeate a reboot?
Spring Data Elasticsearch Bulk Index/Delete - Millions of Records
How to filter map content by path

Trending Discussions on elasticsearch

QUESTION

Why should I run multiple elasticsearch nodes on a single docker host?

Asked 2022-Mar-09 at 18:17

There are a lot of articles online about running an Elasticsearch multi-node cluster using docker-compose, including the official documentation for Elasticsearch 8.0. However, I cannot find a reason why you would set up multiple nodes on the same docker host. Is this the recommended setup for a production environment? Or is it an example of theory in practice?

ANSWER

Answered 2022-Mar-04 at 15:49

You shouldn't consider this a production environment. The guides are examples, often for lab environments, and testing scenarios with the application. I would not consider them production ready, and compose is often not considered a production grade tool since everything it does is to a single docker node, where in production you typically want multiple nodes spread across multiple availability zones.

Source https://stackoverflow.com/questions/71298259

QUESTION

EFK system is build on docker but fluentd can't start up

Asked 2022-Feb-27 at 16:59

I want to build the efk logger system by docker compose. Everything is setup, only fluentd has problem.

fluentd docker container logs

2022-02-15 02:06:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

2022-02-15 02:06:11 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'

2022-02-15 02:06:11 +0000 [info]: gem 'fluentd' version '1.12.0'

/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

my directory:

 my project/
 ├─ fluentd/
 │  ├─ conf/
 │  │  └── fluent.conf
 │  └── Dockerfile
 └── docker-compose.yml

docker-compose.yml:

version: "3"
services:
  web:
    image: httpd
    ports:
      - "8010:80"
    depends_on:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: 127.0.0.1:24224
        fluentd-async: 'true'
        tag: httpd.access

  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.1
    environment:
      - discovery.type=single-node
    expose:
      - 9200
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.1
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"

Dockerfile:

# fluentd/Dockerfile

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"]
USER fluent

fluent.conf:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>

  <store>
    @type stdout
  </store>
</match>

ANSWER

Answered 2022-Feb-15 at 11:35

I faced the same problem, but I used to make exactly the same image where everything works to this day. I can't figure out what has changed.

But if you need to urgently solve the problem, use my in-person image:

docker pull kurraj/fluentd_castom:latest

my dockerfile:

FROM fluent/fluentd:v1.14-1

USER root

RUN gem update --system && \
gem install fluent-plugin-elasticsearch --source http://rubygems.org

Source https://stackoverflow.com/questions/71120621

QUESTION

How to specify schema for nested json in flask_restx?

Asked 2022-Jan-09 at 20:53

Im using flask_restx for swagger API's. The versions are as follows:

python - 3.8.0
flask  - 2.0.2
flask_restx - 0.5.1

The following is the nested json I need to specify the schema for:

dr_status_fields = app_api.model('DR Status',{
    fields.String: {
        "elasticsearch": {
            "backup_status": fields.String,
            "backup_folder": fields.String,
        },
        "mongodb": {
            "backup_status": fields.String,
            "backup_folder": fields.String,
        },
        "postgresdb": {
            "backup_status": fields.String,
            "backup_folder": fields.String,
        },
        "overall_backup_status": fields.String
    }
})

But with this when I load the swagger URL in browser , i get error in the command line:

Unable to render schema
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask_restx/api.py", line 571, in __schema__
    self._schema = Swagger(self).as_dict()
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 268, in as_dict
    "definitions": self.serialize_definitions() or None,
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 623, in serialize_definitions
    return dict(
  File "/usr/local/lib/python3.8/site-packages/flask_restx/swagger.py", line 624, in <genexpr>
    (name, model.__schema__)
  File "/usr/local/lib/python3.8/site-packages/flask_restx/model.py", line 76, in __schema__
    schema = self._schema
  File "/usr/local/lib/python3.8/site-packages/flask_restx/model.py", line 159, in _schema
    properties[name] = field.__schema__
AttributeError: 'dict' object has no attribute '__schema__'

I have tried checking the documentaion of flask_restx for any sample example usage. But could not find any. Please help regarding the same

Update

As per r-m-n answer i have tried to apply the same. But here the problem is there is a key i.e in Line No.2 of the above defined scheme -> fields.String: { . I think this is causing the issue. I have tried to use it as follows:

dr_status_fields = app_api.model('Backup and restore related fields', {
            "backup_status": fields.String,
            "backup_folder": fields.String,
})

dr_db_fields = app_api.model('DR Status', {
    "elasticsearch": fields.Nested(dr_status_fields),
    "mongodb": fields.Nested(dr_status_fields),
    "postgresdb": fields.Nested(dr_status_fields),
    "overall_backup_status": fields.String
})

dr_timestamp_field = app_api.model('Timestamp', {
    fields.String: fields.Nested(dr_db_fields)
})

Here the dr_timestamp_field has resulted in the same issue.

ANSWER

Answered 2022-Jan-09 at 15:27

use fields.Nested:

backup_fields = api.model('Backup fields', {
    "backup_status": fields.String,
    "backup_folder": fields.String
})

dr_status_fields = api.model('DR Status', {
    "elasticsearch": fields.Nested(backup_fields),
    "mongodb": fields.Nested(backup_fields),
    "postgresdb": fields.Nested(backup_fields),
    "overall_backup_status": fields.String
})

Source https://stackoverflow.com/questions/70594476

QUESTION

ElasticSearch - calling UpdateByQuery and Update in parallel causes 409 conflicts

Asked 2021-Dec-19 at 12:15

Working with large index (100,000 documents), I have a use case that spawns several threads who are trying to update documents in parallel, the source code uses two methods in updating the documents: Update and UpdateByQuery, so some of the threads calls Update and some of them calls UpdateByQuery. For the sake of brevity, each of the threads tries to update the same property for the entire documents.

Here is a small POC that demonstrate the use case:

Indexing 100,000 documents of Product type, and spawn 100 tasks so each task calls Update and UpdateByQuery in parallel. both of them uses MatchAll query.

public async Task ConflictsTestAsync()
{
    // index 100,000 documents.
    IEnumerable<Product> products = CreateProducts(100000);
    await _client.IndexManyAsync(products);

    await _client.Indices.UpdateSettingsAsync("MyIndex", s => s
        .IndexSettings(i => i.Setting(UpdatableIndexSettings.MaxResultWindow, 100000)));

    var searchResponse = _client.Search<Product>(s => s
        .From(0)
        .Size(100000)
        .Query(q => q.MatchAll())
    );
    IReadOnlyCollection<IHit<Product>> getProducts = searchResponse.Hits;

    List<Task> tasks = new List<Task>();
    for (int i = 0; i < 100; i++)
    {
        tasks.Add(UpdateByQuery(getProducts));
        tasks.Add(Update(getProducts));
    }

    await Task.WhenAll(tasks);
}

public class Product
{
    public string Price { get; set; }
    public string Id { get; set; }
}

UpdateByQuery (update the price to be 0):

private async Task UpdateByQuery()
{
    Func<UpdateByQueryDescriptor<Product>, IUpdateByQueryRequest> updateByQuerySelector = (UpdateByQueryDescriptor<Product> updateByQueryDescriptor) =>
    {
        updateByQueryDescriptor
            .Conflicts(Conflicts.Abort)
            .ErrorTrace()
            .Query(x => x.MatchAll())
            .Script(x => x.Source("ctx._source['price'] = '0'"));               

        IUpdateByQueryRequest result = updateByQueryDescriptor;
        return result;
    };

    await _client.UpdateByQueryAsync(updateByQuerySelector, CancellationToken.None);
}

Update (update the price to be 1):

private async Task Update(IReadOnlyCollection<IHit<Product>> keys)
{
    foreach (IHit<Product> product in keys)
    {
        DocumentPath<Product> id = new DocumentPath<Product>(product.Id);
        Func<UpdateDescriptor<Product, Product>, IUpdateRequest<Product, Product>> updateSelector = (UpdateDescriptor<Product, Product> updateDescriptor) =>
        {
            var page = product.Source;
            page.Price = "1";

            updateDescriptor.Doc(page);

            IUpdateRequest<Product, Product> result = updateDescriptor;

            return result;
        };

        await _client.UpdateAsync<Product>(id, updateSelector);
    }
}

The problem is spawning multiple threads of Query and UpdateByQuery causes conflicts exceptions:

Some of the Update throws:

Invalid NEST response built from a unsuccessful (409) low level call on POST: /MyIndex/_update/d75a34ae-2533-4e15-a852-13e98c5b599c Audit trail of this API call:

[1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.2495465 OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 409 from: POST /MyIndex/_update/d75a34ae-2533-4e15-a852-13e98c5b599c. ServerError: Type: version_conflict_engine_exception Reason: "[d75a34ae-2533-4e15-a852-13e98c5b599c]: version conflict, required seqNo [701069], primary term [1]. current document has seqNo [702042] and primary term [1]" Request: {"doc":{"id":"d75a34ae-2533-4e15-a852-13e98c5b599c","manufacturer":"777e1602-8390-40c8-817e-fdef4e3fb9c0","price":"1","title":"31184e90-f1d1-45be-8746-496a50de2f97","description":"780cc1ab-0a8b-4114-a840-67a528de8e55"}} Response: {"error":{"root_cause":[{"type":"version_conflict_engine_exception","reason":"[d75a34ae-2533-4e15-a852-13e98c5b599c]: version conflict, required seqNo [701069], primary term [1]. current document has seqNo [702042] and primary term [1]","index_uuid":"gFAfYHfNQZG0tD0L2a9yag","shard":"0","index":"MyIndex"}],"type":"version_conflict_engine_exception","reason":"[d75a34ae-2533-4e15-a852-13e98c5b599c]: version conflict, required seqNo [701069], primary term [1]. current document has seqNo [702042] and primary term [1]","index_uuid":"gFAfYHfNQZG0tD0L2a9yag","shard":"0","index":"MyIndex"},"status":409}

And some of the UpdateByQuery throws (I've cleaned the failures array):

Invalid NEST response built from a unsuccessful (409) low level call on POST: /MyIndex/_update_by_query?conflicts=abort&error_trace=true Audit trail of this API call:

[1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.2636546 OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 409 from: POST /MyIndex/_update_by_query?conflicts=abort&error_trace=true Request: {"query":{"match_all":{}},"script":{"source":"ctx._source['price'] = '0'"}} Response: {"took":120,"timed_out":false,"total":100000,"updated":0,"deleted":0,"batches":1,"version_conflicts":1000,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"MyIndex","type":"_doc","id":"d5fb4183-4ff4-43c9-962c-ee9d0ee59a6b","cause":{"type":"version_conflict_engine_exception","reason":"[d5fb4183-4ff4-43c9-962c-ee9d0ee59a6b]: version conflict, required seqNo [0], primary term [1]. current document has seqNo [100000] and primary term [1]","index_uuid":"pPDFKhj6T4y-MpzYECKpxQ","shard":"0"]}

Because performance are important I prefer not to give up on UpdateByQuery (and uses only Update) or mutexing the methods access, any suggestion to deal with this situation will be appreciated.

Update: Data integrity is important: .Conflicts(Conflicts.Abort)

ElasticSearch: 7.10.0.

ANSWER

Answered 2021-Dec-15 at 12:54

TL;DR: You can pass conflicts=proceed to the update_by_query API if you want it to continue working even when hitting conflicts.

More details: The update_by_query page explains:

When you submit an update by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and updates matching documents using internal versioning. When the versions match, the document is updated and the version number is incremented. If a document changes between the time that the snapshot is taken and the update operation is processed, it results in a version conflict and the operation fails. You can opt to count version conflicts instead of halting and returning by setting conflicts to proceed.

So basically, your update and update_by_query are trying to update the same documents, conflicting with each other. Using conflicts=proceed makes that operation say "oh well, I'll just continue to update the other docs".

Source https://stackoverflow.com/questions/70362695

QUESTION

Using Docker-Desktop for Windows, how can sysctl parameters be configured to permeate a reboot?

Asked 2021-Sep-29 at 12:33

Running elasticsearch in win10 [wsl2] docker-desktop requires to increase mmap counts to 262144 through sysctl -w vm.max_map_count=262144

C:\Users\<username>>wsl -d docker-desktop
<hostname>:/tmp/docker-desktop-root/mnt/host/c/Users/<username># sysctl -w vm.max_map_count=262144

In an answer to this question and elastic's doc it is suggested to accordingly update the vm.max_map_count in /etc/sysctl.conf to set this value permanent, i. e. making it survive the reboot.

Please note the output below, showing the effective mmap count did not adapt to the value defined in sysctl.conf, after my attempt to follow this suggestion and after reboot.

Why didn't it?

PS C:\Users\<username>> wsl -d docker-desktop
<hostname>:/tmp/docker-desktop-root/mnt/host/c/Users/<username># sysctl vm.max_map_count
vm.max_map_count = 65530
<hostname>:/tmp/docker-desktop-root/mnt/host/c/Users/<username># cd /etc
<hostname>:/etc# cat sysctl.conf
# content of this file will override /etc/sysctl.d/*
vm.max_map_count=262144

Considering to have changed the wrong sysctl.conf in etc/ I tried to change the sysctl.conf in <hostname>:/tmp/docker-desktop-root/etc#. This was not allowed as this is a Read-only file system.

Can/should this be parameterized in .wslconfig instead, if so, how?

Using openSUSE Leap 15.2 instead of Win10's PowerShell did not change the outcome.

<username@hostname>:~> sysctl vm.max_map_count
vm.max_map_count = 65530
<username@hostname>:~> cat /etc/sysctl.conf
####
#
# /etc/sysctl.conf is meant for local sysctl settings
#
# sysctl reads settings from the following locations:
#   /boot/sysctl.conf-<kernelversion>
#   /lib/sysctl.d/*.conf
#   /usr/lib/sysctl.d/*.conf
#   /usr/local/lib/sysctl.d/*.conf
#   /etc/sysctl.d/*.conf
#   /run/sysctl.d/*.conf
#   /etc/sysctl.conf
#
# To disable or override a distribution provided file just place a
# file with the same name in /etc/sysctl.d/
#
# See sysctl.conf(5), sysctl.d(5) and sysctl(8) for more information
#
####
vm.max_map_count=262144

Notably, the header in the above sysctl.conf file refers to a /boot/sysctl.conf-<kernelversion>. Taking into account, mmap count might need to be defined in this file in order to be considered when booting, I tried to set the required parameter there, but had no permission to do so.

<username@hostname>:/boot> sudo echo 'vm.max_map_count=262144' >> sysctl.conf-5.10.16.3-microsoft-standard-WSL2
-bash: sysctl.conf-5.10.16.3-microsoft-standard-WSL2: Permission denied

One of the known issues and limitations reported in openSUSE:WSL - openSUSE Wiki is

A system in WSL does not actually boot and does not use systemd. A proprietary Microsoft /init binary initializes the system. Therefore service management does not work like in a VM. It rather behaves like an interactive container.

Can anyone verify this to be the reason why parameterizing sysctl.conf cannot be an effective approach to the purpose at hand?

System

Host
OS: Microsoft Windows 10 Enterprise
Version: 10.0.18363 Build 18363 \

Docker
Client:
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:02 2021
OS/Arch: linux/amd64
Plugins:
compose: Docker Compose (Docker Inc., v2.0.0-rc.2)
scan: Docker Scan (Docker Inc., v0.8.0)

Server:
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:10 2021
OS/Arch: linux/amd64
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0 \

Kernel Version: 5.10.16.3-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 12.39GiB
Name: docker-desktop

ANSWER

Answered 2021-Sep-29 at 12:33

Short answer:

In your Windows %userprofile% directory (typically C:\Users\<username>) create or edit the file .wslconfig with the following:

[wsl2]
kernelCommandLine = "sysctl.vm.max_map_count=262144"

Then exit any WSL instance, wsl --shutdown, and restart.

> sysctl vm.max_map_count
vm.max_map_count = 262144

More Detail:

You really have three (closely related) questions:

(Mainly) How to best persist this setting
Can it be done via .wslconfig?

Can/should this be parameterized in .wslconfig instead, if so, how?
Why isn't /etc/sysctl.conf working to persist it?

/etc/sysctl.conf to set this value permanent, i. e. making it survive the reboot .... Why didn't it?

We'll combine the first two, since the answer is "Yes, you can do it via .wslconfig.

This wasn't easy to find, since the change is fairly recent. It comes curtesy this answer.

Starting with kernel release 5.8, you can set sysctl values on the kernel command-line using the sysctl.* prefix to the value.

So as long as you have upgraded your WSL kernel (which I had not on two of my systems since wsl --update only recently was released in 21H1), you can set your %userprofile%\.wslconfig as above.

/etc/sysctl.conf

/etc/sysctl.conf to set this value permanent, i. e. making it survive the reboot .... Why didn't it?

There's a lot more to this section than is really needed since I went down this rabbit trail first and posted it as the original answer. But first, "why" it wasn't working ...

This is a known issue of sorts with WSL2.

While sysctl and Systemd/systemctl are two completely different things, in that issue therealkenc (of the Microsoft WSL team) points out that something must read and process /etc/sysctl.conf. In most Linux distributions, that's Systemd or the SysVInit scripts.

WSL, on the other, uses it's own /init in place of SysVInit or Systemd. The WSL init process does not (currently) have a mechanism for running services at startup, as you found in the openSUSE doc. So ... nothing reads and acts on /etc/sysctl.conf at startup.

The remainder of this answer includes some systcl.conf workarounds that aren't needed given we can do this with .wslconfig. This is left in just in case anyone finds it useful for some reason.

There are a few workarounds for this. All of them are going to involve finding a way to run sysctl -w vm.max_map_count=262144 (as root) automatically:

First, in Windows 11, WSL has a new feature to simplify this. Simply create/edit /etc/wsl.conf (via sudo) with:

[boot]
command="sysctl -w vm.max_map_count=262144"

This will run at root each time the WSL instance starts. As far as I know, this will also work from within the docker-desktop instance, but I have not been able to test that.

Important Note: Please back up your WSL instance using wsl --export, or at least backup your Docker volumes, images, and containers, before attempting this change. It's my experience that it is possible via a stalled boot command to completely prevent the instance from booting, with no recovery mechanism that I have (yet) found. Fortunately, I was working in a newly installed test instance, so I was able to wsl --unregister it (completely destroying it) without losing anything other than a few minutes.

For the many of us who will be on Windows 10 for some time to come, there are several options. Unfortunately I haven't been able to test these against Docker, but I've used similar techniques with other WSL instances in the past:

If Docker Desktop is set to run when you log in, then you could add a scheduled task (via Windows Task Scheduler) to also run at login. This would be a:
- "Basic Task"
- With the "Program" set to wsl.exe
- And the Arguments set to -d docker-desktop sh -c "sysctl -w vm.max_map_count=262144"
Note that if the docker-desktop WSL instance terminates at any point, then the change will be lost until next reboot. However, because Docker Desktop is running at boot/login, the instance should remain up with this change in place.

Depending on your workflow, this may or may not fit. If you are using a separate WSL instance (e.g. Ubuntu) for development purposes, you can have that instance modify the docker-desktop instance when you start up.

So that you don't have to type the sudo password each time you start up, use the following line in your .bashrc (or equivalent):

wsl.exe -d docker-desktop sh -c "sysctl -w vm.max_map_count=262144"

One final suggestion -- Submit a feature request to the Docker Desktop team. It would, IMHO, be easiest for Docker Desktop's startup process to read /etc/sysctl.conf and process it accordingly.

Source https://stackoverflow.com/questions/69214301

QUESTION

Spring Data Elasticsearch Bulk Index/Delete - Millions of Records

Asked 2021-Sep-29 at 09:43

I'm using Spring Data Elasticsearch 4.2.5, we have a job that does ETL (extract, transform and load data) to a particular database table. I'm indexing this data using Elasticsearch while the job is running. The data will be in millions of records and more. Currently, I'm doing index on every iteration. I read that, using elasticsearch index on every iteration might take some time. I wanted to use something like bulk-index, but for that I need to add indexQuery object to List. Adding millions of records to list and doing bulk-index may bring memory issues.

I need to apply similar kind of process for deletion. When records are deleted based on some common ID, I need to delete related elastic documents and this will also be in millions and more.

Is there anyway to do indexing/deleting very fast for this requirement? Any help is much appreciated and correct me if my understanding is incorrect.

INDEXING

for (Map.Entry<Integer, ObjectDetails> key : objectDetailsHashMap.entrySet()) {
    indexDocument(elasticsearchOperations, key, oPath);
    // other code to insert data in db table...
 }

private void indexDocument(ElasticsearchOperations elasticsearchOperations,
                              Map.Entry<Integer, ObjectDetails> key, String oPath) {
    String docId = "" + key.getValue().getCatalogId() + key.getValue().getObjectId();

    byte[] nameBytes = key.getValue().getName();
    byte[] physicalNameBytes = key.getValue().getPhysicalName();
    byte[] definitionBytes =  key.getValue().getDefinition();
    byte[] commentBytes = key.getValue().getComment();

    IndexQuery indexQuery = new IndexQueryBuilder()
            .withId(docId)
            .withObject(new MetadataSearch(
                    key.getValue().getObjectId(),
                    key.getValue().getCatalogId(),
                    key.getValue().getParentId(),
                    key.getValue().getTypeCode(),
                    key.getValue().getStartVersion(),
                    key.getValue().getEndVersion(),
                    nameBytes != null ? new String(nameBytes, StandardCharsets.UTF_8) : "-",
                    physicalNameBytes != null ? new String(physicalNameBytes, StandardCharsets.UTF_8) : "-",
                    definitionBytes != null ? new String(definitionBytes, StandardCharsets.UTF_8) : "-",
                    commentBytes != null ? new String(commentBytes, StandardCharsets.UTF_8) : "-",
                    oPath
            ))
            .build();

    elasticsearchOperations.index(indexQuery, IndexCoordinates.of("portal_idx"));
}

DELETING

private void deleteElasticDocuments(String catalogId) {
    String queryText = martServerContext.getQueryCacheInstance().getQuery(QUERY_PORTAL_GET_OBJECTS_IN_PORTAL_BY_MODEL);
    MapSqlParameterSource mapSqlParameterSource = new MapSqlParameterSource();
    mapSqlParameterSource.addValue("cId", Integer.parseInt(catalogId));
    namedParameterJdbcTemplate.query(queryText, mapSqlParameterSource, (resultSet -> {
        int objectId = resultSet.getInt(O_ID);
        String docId = catalogId + objectId;
        elasticsearchOperations.delete(docId, IndexCoordinates.of("portal_idx"));
    }));
}

ANSWER

Answered 2021-Sep-29 at 09:43

For adding the documents you could use bulk indexing for example by collecting the documents to index in a list/array or whatever and when a predefined size is reached - like 500 entries - then do a bulk insert of these.

For deleting there is no bulk operation, but you could collect ids to delete in a list or array again with am maximum size and then use ElasticsearchOperations.idsQuery(List<String>) to create a query for these ids and pass this into the delete(query) method.

Edit 29.09.2021:

the idsQuery was just added in the 4.3 branch, it is simplemented like this (https://github.com/spring-projects/spring-data-elasticsearch/blob/main/src/main/java/org/springframework/data/elasticsearch/core/AbstractElasticsearchRestTransportTemplate.java#L193-L200):

@Override
public Query idsQuery(List<String> ids) {

    Assert.notNull(ids, "ids must not be null");

    return new NativeSearchQueryBuilder().withQuery(QueryBuilders.idsQuery().addIds(ids.toArray(new String[] {})))
            .build();
}

Source https://stackoverflow.com/questions/69362887

QUESTION

How to filter map content by path

Asked 2021-Sep-05 at 02:10

I want to select paths of a deeply nested map to keep.

For example:

{:a 1
 :b {:c [{:d 1 :e 1} 
         {:d 2 :e 2}]
     :f 1}
 :g {:h {:i 4 :j [1 2 3]}}}

I want to select by paths, like so:

(select-paths m [[:a] 
                 [:b :c :e]
                 [:b :f]
                 [:g :h :i]])

This would return

{:a 1
 :b {:c [{:e 1}
         {:e 2}]
     :f 1}
 :g {:h {:i 4}}}

Essentially the same as Elasticsearch's fields parameter. The format of the paths argument can be something else, this is just the first idea.

I tried two different solutions

Go through the entire map and checking if the full path of the current element is in the given paths. I can't figure out how to handle lists of maps so that they are kept as lists of maps.
Creating select-keys statements from the given paths but again I run into problems with lists of maps - and especially trying to resolve paths of varying depths that have some common depth.

I looked at spectre but I didn't see anything that would do this. Any map or postwalk based solution I come up with turns into something incredibly convoluted at some point. I must be thinking about this the wrong way.

If there's a way to do this with raw json, that would be fine as well. Or even a Java solution.

ANSWER

Answered 2021-Sep-03 at 17:18

There is no simple way to accomplish your goal. The automatic processing implied for the sequence under [:b :c] is also problematic.

You can get partway there using the Tupelo Forest library. See the Lightning Talk video from Clojure/Conj 2017.

I did some additional work in data destructuring that you may find useful building the tupelo.core/destruct macro (see examples here). You could follow a similar outline to build a recursive solution to your specific problem.

A related project is Meander. I have worked on my own version which is like a generalized version of tupelo.core/destruct. Given data like this

(def skynet-widgets [{:basic-info   {:producer-code "Cyberdyne"}
                      :widgets      [{:widget-code      "Model-101"
                                      :widget-type-code "t800"}
                                     {:widget-code      "Model-102"
                                      :widget-type-code "t800"}
                                     {:widget-code      "Model-201"
                                      :widget-type-code "t1000"}]
                      :widget-types [{:widget-type-code "t800"
                                      :description      "Resistance Infiltrator"}
                                     {:widget-type-code "t1000"
                                      :description      "Mimetic polyalloy"}]}
                     {:basic-info   {:producer-code "ACME"}
                      :widgets      [{:widget-code      "Dynamite"
                                      :widget-type-code "c40"}]
                      :widget-types [{:widget-type-code "c40"
                                      :description      "Boom!"}]}])

You can search and extract data using a template like this:


    (let [root-eid (td/add-entity-edn skynet-widgets)
          results  (td/match
                     [{:basic-info   {:producer-code ?}
                       :widgets      [{:widget-code      ?
                                       :widget-type-code wtc}]
                       :widget-types [{:widget-type-code wtc
                                       :description      ?}]}])]
      (is= results
        [{:description "Resistance Infiltrator" :widget-code "Model-101" :producer-code "Cyberdyne" :wtc "t800"}
         {:description "Resistance Infiltrator" :widget-code "Model-102" :producer-code "Cyberdyne" :wtc "t800"}
         {:description "Mimetic polyalloy" :widget-code "Model-201" :producer-code "Cyberdyne" :wtc "t1000"}
         {:description "Boom!" :widget-code "Dynamite" :producer-code "ACME" :wtc "c40"}])))

This code is working (see here) but it needs more polish. You could use it as a guide to building a generalized select-paths function.

Can you add any details on how this problem arose or the specific context? That may point to ideas for an alternate solution.

Source https://stackoverflow.com/questions/69047488

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install elasticsearch

The simplest way to set up Elasticsearch is to create a managed deployment with Elasticsearch Service on Elastic Cloud. If you prefer to install and manage Elasticsearch yourself, you can download the latest version from elastic.co/downloads/elasticsearch. For more installation options, see the Elasticsearch installation documentation.
To upgrade from an earlier version of Elasticsearch, see the Elasticsearch upgrade documentation.
Elasticsearch uses Gradle for its build system.