kandi background
Explore Kits

elasticsearch-jdbc | JDBC importer for Elasticsearch | DB Client library

 by   jprante Java Version: 2.0.0.0 License: Non-SPDX

 by   jprante Java Version: 2.0.0.0 License: Non-SPDX

Download this library from

kandi X-RAY | elasticsearch-jdbc Summary

elasticsearch-jdbc is a Java library typically used in Utilities, DB Client applications. elasticsearch-jdbc has no bugs, it has no vulnerabilities, it has build file available and it has medium support. However elasticsearch-jdbc has a Non-SPDX License. You can download it from GitHub.
The Java Database Connection (JDBC) importer allows to fetch data from JDBC sources for indexing into Elasticsearch. The JDBC importer was designed for tabular data. If you have tables with many joins, the JDBC importer is limited in the way to reconstruct deeply nested objects to JSON and process object semantics like object identity. Though it would be possible to extend the JDBC importer with a mapping feature where all the object properties could be specified, the current solution is focused on rather simple tabular data streams. Assuming you have a table of name orders with a primary key in column id, you can issue this from the command line. And that's it. Now you can check your Elasticsearch cluster for the index jdbc or your Elasticsearch logs about what happened.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • elasticsearch-jdbc has a medium active ecosystem.
  • It has 2834 star(s) with 719 fork(s). There are 233 watchers for this library.
  • It had no major release in the last 12 months.
  • There are 401 open issues and 478 have been closed. On average issues are closed in 284 days. There are 18 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of elasticsearch-jdbc is 2.0.0.0
elasticsearch-jdbc Support
Best in #DB Client
Average in #DB Client
elasticsearch-jdbc Support
Best in #DB Client
Average in #DB Client

quality kandi Quality

  • elasticsearch-jdbc has 0 bugs and 0 code smells.
elasticsearch-jdbc Quality
Best in #DB Client
Average in #DB Client
elasticsearch-jdbc Quality
Best in #DB Client
Average in #DB Client

securitySecurity

  • elasticsearch-jdbc has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • elasticsearch-jdbc code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.
elasticsearch-jdbc Security
Best in #DB Client
Average in #DB Client
elasticsearch-jdbc Security
Best in #DB Client
Average in #DB Client

license License

  • elasticsearch-jdbc has a Non-SPDX License.
  • Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.
elasticsearch-jdbc License
Best in #DB Client
Average in #DB Client
elasticsearch-jdbc License
Best in #DB Client
Average in #DB Client

buildReuse

  • elasticsearch-jdbc releases are available to install and integrate.
  • Build file is available. You can build the component from source.
  • Installation instructions, examples and code snippets are available.
  • elasticsearch-jdbc saves you 6189 person hours of effort in developing the same functionality from scratch.
  • It has 12895 lines of code, 953 functions and 117 files.
  • It has high code complexity. Code complexity directly impacts maintainability of the code.
elasticsearch-jdbc Reuse
Best in #DB Client
Average in #DB Client
elasticsearch-jdbc Reuse
Best in #DB Client
Average in #DB Client
Top functions reviewed by kandi - BETA

kandi has reviewed elasticsearch-jdbc and discovered the below as its top functions. This is intended to give you an instant insight into elasticsearch-jdbc implemented functionality, and help decide if they suit your requirements.

  • Parses the JDBC type .
    • Returns the time after the given date
      • Formats a period in milliseconds .
        • Sets the current structured object values .
          • Starts the executor .
            • Writes the metrics to a file .
              • Get the op infos .
                • Creates a new index .
                  • Writes the metrics .
                    • Converts a locale string to a Locale object .

                      Get all kandi verified functions for this library.

                      Get all kandi verified functions for this library.

                      elasticsearch-jdbc Key Features

                      JDBC importer for Elasticsearch

                      JDBC importer for Elasticsearch

                      copy iconCopydownload iconDownload
                      bin=$JDBC_IMPORTER_HOME/bin
                      lib=$JDBC_IMPORTER_HOME/lib
                      echo '{
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "user" : "",
                              "password" : "",
                              "sql" : "select *, id as _id from orders"
                          }
                      }' | java \
                             -cp "${lib}/*" \
                             -Dlog4j.configurationFile=${bin}/log4j2.xml \
                             org.xbib.tools.Runner \
                             org.xbib.tools.JDBCImporter
                      

                      JDBC importer definition file

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                               <definition>
                          }
                      }
                      

                      Parameters

                      copy iconCopydownload iconDownload
                      "sql" : [
                          {
                              "statement" : "select ... from ... where a = ?, b = ?, c = ?",
                              "parameter" : [ "value for a", "value for b", "value for c" ]
                          },
                          {
                              "statement" : "insert into  ... where a = ?, b = ?, c = ?",
                              "parameter" : [ "value for a", "value for b", "value for c" ],
                              "write" : "true"
                          },
                          {
                              "statement" : ...
                          }
                      ]
                      

                      Overview about the default parameter settings

                      copy iconCopydownload iconDownload
                      {
                          "jdbc" : {
                      		"strategy" : "standard",
                              "url" : null,
                              "user" : null,
                              "password" : null,
                              "sql" : null,
                              "locale" : /* equivalent to Locale.getDefault().toLanguageTag() */,
                              "timezone" : /* equivalent to TimeZone.getDefault() */,
                              "rounding" : null,
                              "scale" : 2,
                              "autocommit" : false,
                              "fetchsize" : 10, /* if URL contains MySQL JDBC driver URL, this is Integer.MIN */
                              "max_rows" : 0,
                              "max_retries" : 3,
                              "max_retries_wait" : "30s",
                              "resultset_type" : "TYPE_FORWARD_ONLY",
                              "resultset_concurreny" : "CONCUR_UPDATABLE",
                              "ignore_null_values" : false,
                              "prepare_database_metadata" : false,
                              "prepare_resultset_metadata" : false,
                              "column_name_map" : null,
                              "query_timeout" : 1800,
                              "connection_properties" : null,
                      		"schedule" : null,
                      		"interval" : 0L,
                      		"threadpoolsize" : 1,
                              "index" : "jdbc",
                              "type" : "jdbc",
                              "index_settings" : null,
                              "type_mapping" : null,
                      		"max_bulk_actions" : 10000,
                      		"max_concurrent_bulk_requests" : 2 * available CPU cores,
                      		"max_bulk_volume" : "10m",
                      		"max_request_wait" : "60s",
                      		"flush_interval" : "5s"
                          }
                      }
                      

                      Time scheduled execution

                      copy iconCopydownload iconDownload
                          "schedule" : "0 0-59 0-23 ? * *"
                      

                      Structured objects

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "user" : "",
                              "password" : "",
                              "sql" : "select \"relations\" as \"_index\", orders.customer as \"_id\", orders.customer as \"contact.customer\", employees.name as \"contact.employee\" from orders left join employees on employees.department = orders.department order by _id"
                          }
                      }
                      

                      Column names for JSON document construction

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "user" : "",
                              "password" : "",
                              "sql" : "select products.name as \"product.name\", orders.customer as \"product.customer.name\", orders.quantity * products.price as \"product.customer.bill\" from products, orders where products.name = orders.product"
                          }
                      }
                      

                      Bracket notation for JSON array construction

                      copy iconCopydownload iconDownload
                      {
                          "blog" : {
                              "attachment": [
                                  {
                                      "name" : "/web/q/g/h/57436356.jpg",
                                      "id" : "9450"
                                  },
                                  {
                                      "name" : "/web/i/s/q/GS3193626.jpg",
                                      "id" : "9965"
                                  },
                                  {
                                      "name" : "/web/i/s/q/GS3193626.jpg",
                                      "id" : "9451"
                                  }
                              ],
                              "name" : "Joe",
                              "association" : [
                                  {
                                      "name" : "John",
                                      "id" : "3917"
                                  },
                                  {
                                      "name" : "John",
                                      "id" : "3917"
                                  },
                                  {
                                      "name" : "John",
                                      "id" : "3917"
                                  }
                               ],
                               "published":"2014-01-06 00:00:00"
                           }
                      }
                      

                      How to fetch a table?

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "user" : "",
                              "password" : "",
                              "sql" : "select * from orders"
                          }
                      }
                      

                      How to update a table?

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "user" : "",
                              "password" : "",
                              "sql" : [
                                  {
                                      "statement" : "select * from \"products\""
                                  },
                                  {
                                      "statement" : "delete from \"products\" where \"_job\" = ?",
                                      "parameter" : [ "$job" ]
                                  }
                              ],
                              "index" : "my_jdbc_index",
                              "type" : "my_jdbc_type"
                          }
                      }
                      

                      How to select incremental data from a table?

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:mysql://localhost:3306/test",
                              "statefile" : "statefile.json",
                              "user" : "",
                              "password" : "",
                              "sql" : [
                                  {
                                      "statement" : "select * from products where mytimestamp > ?",
                                      "parameter" : [ "$metrics.lastexecutionstart" ]
                                  }
                              ],
                              "index" : "my_jdbc_index",
                              "type" : "my_jdbc_type"
                          }
                      }
                      

                      Stored procedures or callable statements

                      copy iconCopydownload iconDownload
                      create procedure GET_SUPPLIER_OF_COFFEE(
                          IN coffeeName varchar(32), 
                          OUT supplierName varchar(40)) 
                          begin 
                              select SUPPLIERS.SUP_NAME into supplierName 
                              from SUPPLIERS, COFFEES 
                              where SUPPLIERS.SUP_ID = COFFEES.SUP_ID 
                              and coffeeName = COFFEES.COF_NAME; 
                              select supplierName; 
                          end
                      

                      How to import from a CSV file?

                      copy iconCopydownload iconDownload
                      {
                      	"type" : "jdbc",
                      	"jdbc" : {
                      		"driver" : "org.xbib.jdbc.csv.CsvDriver",
                      		"url" : "jdbc:xbib:csv:mydatadir?columnTypes=&separator=,",
                      		"user" : "",
                      		"password" : "",
                      		"sql" : "select * from mycsvfile"
                      	}
                      }
                      

                      Persisted state

                      copy iconCopydownload iconDownload
                      "sql" : ...,
                      "statefile" : "statefile.json",
                      ...
                      

                      Monitoring the JDBC importer

                      copy iconCopydownload iconDownload
                      "sql" : ...,
                      "schedule" : ...,
                      "statefile" : "statefile.json",
                      "metrics" : {
                          "enabled" : true,
                          "interval" : "1m",
                          "logger" : {
                              "plain" : false,
                              "json" : true
                          }
                      }
                      

                      PostgreSQL

                      copy iconCopydownload iconDownload
                       cp postgresql-9.1-902.jdbc4.jar $JDBC_IMPORTER_HOME/lib
                      

                      MS SQL Server

                      copy iconCopydownload iconDownload
                      ./elasticsearch.bat
                      

                      Index simple geo coordinates from MySQL in Elasticsearch

                      copy iconCopydownload iconDownload
                      curl -XDELETE 'localhost:9200/myjdbc'
                      bin=$JDBC_IMPORTER_HOME/bin
                      lib=$JDBC_IMPORTER_HOME/lib
                      echo '
                      {
                           "type" : "jdbc",
                           "jdbc" : {
                               "url" : "jdbc:mysql://localhost:3306/test",
                               "user" : "",
                               "password" : "",
                               "locale" : "en_US",
                               "sql" : [
                                   {
                                       "statement" : "select \"myjdbc\" as _index, \"mytype\" as _type, name as _id, city, zip, address, lat as \"location.lat\", lon as \"location.lon\" from geo"
                                   }
                               ],
                               "index" : "myjdbc",
                               "type" : "mytype",
                               "index_settings" : {
                                   "index" : {
                                       "number_of_shards" : 1
                                   }
                               },
                               "type_mapping": {
                                   "mytype" : {
                                       "properties" : {
                                           "location" : {
                                               "type" : "geo_point"
                                           }
                                       }
                                   }
                               }
                           }
                      }'  | java \
                                     -cp "${lib}/*" \
                                     -Dlog4j.configurationFile=${bin}/log4j2.xml \
                                     org.xbib.tools.Runner \
                                     org.xbib.tools.JDBCImporter
                      echo "sleeping while importer should run..."
                      sleep 10
                      curl -XGET 'localhost:9200/myjdbc/_refresh'
                      curl -XPOST 'localhost:9200/myjdbc/_search?pretty' -d '
                      {
                         "query": {
                            "filtered": {
                              "query": {
                                 "match_all": {
                                  }
                              },
                              "filter": {
                                  "geo_distance" : {
                                      "distance" : "20km",
                                      "location" : {
                                           "lat" : 51.0,
                                           "lon" : 7.0
                                       }
                                   }
                               }
                            }
                          }
                      }'
                      

                      Index simple geo coordinates from Postgres/PostGIS geometry field in Elasticsearch

                      copy iconCopydownload iconDownload
                      curl -XDELETE 'localhost:9200/myjdbc'
                      bin=$JDBC_IMPORTER_HOME/bin
                      lib=$JDBC_IMPORTER_HOME/lib
                      echo '
                      {
                           "type" : "jdbc",
                           "jdbc" : {
                               "url" : "jdbc:postgres://localhost:5432/test",
                               "user" : "test",
                               "password" : "test",
                               "locale" : "en_GB",
                               "sql" : "select geonameid as _id, name, admin1_code, admin2_code, admin3_code, round(ST_Y(geom)::numeric,8) as \"location.lat\", round(ST_X(geom)::numeric,8) as \"location.lon\" from geo",
                               "index" : "myjdbc",
                               "type" : "mytype",
                               "scale" : 8,
                               "index_settings" : {
                                   "index" : {
                                       "number_of_shards" : 1
                                   }
                               },
                               "type_mapping": {
                                   "mytype" : {
                                       "properties" : {
                                           "location" : {
                                               "type" : "geo_point"
                                           }
                                       }
                                   }
                               }
                           }
                      }'  | java \
                                     -cp "${lib}/*" \
                                     -Dlog4j.configurationFile=${bin}/log4j2.xml \
                                     org.xbib.tools.Runner \
                                     org.xbib.tools.JDBCImporter
                      echo "sleeping while importer should run..."
                      sleep 10
                      curl -XGET 'localhost:9200/myjdbc/_refresh'
                      curl -XPOST 'localhost:9200/myjdbc/_search?pretty' -d '
                      {
                         "query": {
                            "filtered": {
                              "query": {
                                 "match_all": {
                                  }
                              },
                              "filter": {
                                  "geo_distance" : {
                                      "distance" : "20km",
                                      "location" : {
                                           "lat" : 51.477347,
                                           "lon" : -0.000850
                                       }
                                   }
                               }
                            }
                          }
                      }'
                      

                      Geo shapes

                      copy iconCopydownload iconDownload
                      mysql -u root test <<EOT
                      drop table if exists test.geom;
                      create table test.geom (
                      	id integer,
                      	g geometry
                      );
                      set @g = 'POLYGON((0 0,10 0,10 10,0 10,0 0),(5 5,7 5,7 7,5 7, 5 5))';
                      insert into test.geom values (0, GeomFromText(@g));
                      EOT
                      
                      curl -XDELETE 'localhost:9200/myjdbc'
                      echo '
                      {
                      	"type" : "jdbc",
                      	"jdbc" : {
                      		"url" : "jdbc:mysql://localhost:3306/test",
                      		"user" : "",
                      		"password" : "",
                      		"locale" : "en_US",
                      		"sql" : "select \"myjdbc\" as _index, \"mytype\" as _type, id as _id, astext(g) as polygon from geom",
                      		"elasticsearch" : {
                      			 "cluster" : "elasticsearch",
                      			 "host" : "localhost",
                      			 "port" : 9300
                      		},
                      		"index" : "myjdbc",
                      		"type" : "mytype",
                      		"index_settings" : {
                      			"index" : {
                      				"number_of_shards" : 1
                      			}
                      		},
                      		"type_mapping": {
                      			"mytype" : {
                      				"properties" : {
                      					"polygon" : {
                      						"type" : "geo_shape",
                      						"tree" : "quadtree"
                      					}
                      				}
                      			}
                      		}
                      	}
                      }
                      ' | java \
                      	-cp "${lib}/*" \
                      	-Dlog4j.configurationFile=${bin}/log4j2.xml \
                      	org.xbib.tools.Runner \
                      	org.xbib.tools.JDBCImporter
                      

                      Oracle column name 30 character limit

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:oracle:thin:@//localhost/sid",
                              "user" : "user",
                              "password" : "password",
                              "sql" : "select or_id as \"_id\", or_tan as \"o.t\", or_status as \"o.s\", stages.* from orders, stages where or_id = st_or_id and or_seqno = st_seqno",
                              "column_name_map" : {
                                 "o" : "order",
                                 "t" : "transaction_id",
                                 "s" : "status"
                              }
                          }
                      }
                      

                      Connection properties for JDBC driver

                      copy iconCopydownload iconDownload
                      {
                          "type" : "jdbc",
                          "jdbc" : {
                              "url" : "jdbc:oracle:thin:@//localhost:1521/sid",
                              "user" : "user",
                              "password" : "password",
                              "sql" : "select ... from ...",
                              "connection_properties" : {
                                  "oracle.jdbc.TcpNoDelay" : false,
                                  "useFetchSizeWithLongColumn" : false,
                                  "oracle.net.CONNECT_TIMEOUT" : 10000,
                                  "oracle.jdbc.ReadTimeout" : 50000
                              }
                          }
                      }
                      

                      License

                      copy iconCopydownload iconDownload
                      http://www.apache.org/licenses/LICENSE-2.0
                      

                      load dynamic data from MySQL table in ElasticSearch using JDBC driver

                      copy iconCopydownload iconDownload
                      curl -XDELETE 'localhost:9200/event'
                      bin=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin
                      lib=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib
                      echo '{
                      "type":"jdbc",
                      "jdbc":{
                      
                      "url":"jdbc:mysql://localhost:3306/CDFL",
                      "user":"root",
                      "password":"root",
                      "useSSL":"false",
                      "sql":"SELECT * FROM event",
                      "index":"event",
                      "type":"event",
                      "poll" : "6s",
                      "autocommit":"true",
                      "metrics": {
                                  "enabled" : true
                              },
                              "elasticsearch" : {
                                   "cluster" : "servercluster",
                                   "host" : "localhost",
                                   "port" : 9300 
                              } 
                      }
                      }' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
                      echo "sleeping while importer should run..."
                      sleep 10
                      curl -XGET 'localhost:9200/event/_refresh'
                      
                      sh elasticSearch/event.sh
                      
                      curl -XDELETE 'localhost:9200/event'
                      bin=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin
                      lib=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib
                      echo '{
                      "type":"jdbc",
                      "jdbc":{
                      
                      "url":"jdbc:mysql://localhost:3306/CDFL",
                      "user":"root",
                      "password":"root",
                      "useSSL":"false",
                      "sql":"SELECT * FROM event",
                      "index":"event",
                      "type":"event",
                      "poll" : "6s",
                      "autocommit":"true",
                      "metrics": {
                                  "enabled" : true
                              },
                              "elasticsearch" : {
                                   "cluster" : "servercluster",
                                   "host" : "localhost",
                                   "port" : 9300 
                              } 
                      }
                      }' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
                      echo "sleeping while importer should run..."
                      sleep 10
                      curl -XGET 'localhost:9200/event/_refresh'
                      
                      sh elasticSearch/event.sh
                      

                      Elasticsearch query with NEST don't work

                      copy iconCopydownload iconDownload
                      const string ESServer = "http://localhost:9200";
                      ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer))
                          .DefaultIndex("tiky");
                          .MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"))
                          // pass POCO property names through verbatim
                          .DefaultFieldNameInferrer(s => s);
                      
                      ElasticClient client = new ElasticClient(settings);
                      
                      public class Faq
                      {
                          public string Question { get; set; }
                      }
                      
                      var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
                      var defaultIndex = "default-index";
                      var connectionSettings = new ConnectionSettings(pool)
                              .DefaultIndex(defaultIndex)
                              .DefaultFieldNameInferrer(s => s);
                      
                      var client = new ElasticClient(connectionSettings);
                      
                      if (client.IndexExists(defaultIndex).Exists)
                          client.DeleteIndex(defaultIndex);
                      
                      client.CreateIndex(defaultIndex, c => c
                          .Mappings(m => m
                              .Map<Faq>(mm => mm
                                  // let NEST infer mapping from the POCO
                                  .AutoMap()
                                  // override any inferred mappings explicitly
                                  .Properties(p => p
                                      .String(s => s
                                          .Name(n => n.Question)
                                          .Fields(f => f
                                              .String(ss => ss
                                                  .Name("raw")
                                                  .NotAnalyzed()
                                              )
                                          )
                                      )
                                  )
                              )
                          )
                      );   
                      
                      {
                        "mappings": {
                          "faq": {
                            "properties": {
                              "Question": {
                                "type": "string",
                                "fields": {
                                  "raw": {
                                    "type": "string",
                                    "index": "not_analyzed"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                      
                      client.Search<Faq>(s => s
                          .Query(q => q
                              .Term(f => f.Question.Suffix("raw"), "What's the Question?")
                          )
                      );
                      
                      const string ESServer = "http://localhost:9200";
                      ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer))
                          .DefaultIndex("tiky");
                          .MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"))
                          // pass POCO property names through verbatim
                          .DefaultFieldNameInferrer(s => s);
                      
                      ElasticClient client = new ElasticClient(settings);
                      
                      public class Faq
                      {
                          public string Question { get; set; }
                      }
                      
                      var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
                      var defaultIndex = "default-index";
                      var connectionSettings = new ConnectionSettings(pool)
                              .DefaultIndex(defaultIndex)
                              .DefaultFieldNameInferrer(s => s);
                      
                      var client = new ElasticClient(connectionSettings);
                      
                      if (client.IndexExists(defaultIndex).Exists)
                          client.DeleteIndex(defaultIndex);
                      
                      client.CreateIndex(defaultIndex, c => c
                          .Mappings(m => m
                              .Map<Faq>(mm => mm
                                  // let NEST infer mapping from the POCO
                                  .AutoMap()
                                  // override any inferred mappings explicitly
                                  .Properties(p => p
                                      .String(s => s
                                          .Name(n => n.Question)
                                          .Fields(f => f
                                              .String(ss => ss
                                                  .Name("raw")
                                                  .NotAnalyzed()
                                              )
                                          )
                                      )
                                  )
                              )
                          )
                      );   
                      
                      {
                        "mappings": {
                          "faq": {
                            "properties": {
                              "Question": {
                                "type": "string",
                                "fields": {
                                  "raw": {
                                    "type": "string",
                                    "index": "not_analyzed"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                      
                      client.Search<Faq>(s => s
                          .Query(q => q
                              .Term(f => f.Question.Suffix("raw"), "What's the Question?")
                          )
                      );
                      
                      const string ESServer = "http://localhost:9200";
                      ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer))
                          .DefaultIndex("tiky");
                          .MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"))
                          // pass POCO property names through verbatim
                          .DefaultFieldNameInferrer(s => s);
                      
                      ElasticClient client = new ElasticClient(settings);
                      
                      public class Faq
                      {
                          public string Question { get; set; }
                      }
                      
                      var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
                      var defaultIndex = "default-index";
                      var connectionSettings = new ConnectionSettings(pool)
                              .DefaultIndex(defaultIndex)
                              .DefaultFieldNameInferrer(s => s);
                      
                      var client = new ElasticClient(connectionSettings);
                      
                      if (client.IndexExists(defaultIndex).Exists)
                          client.DeleteIndex(defaultIndex);
                      
                      client.CreateIndex(defaultIndex, c => c
                          .Mappings(m => m
                              .Map<Faq>(mm => mm
                                  // let NEST infer mapping from the POCO
                                  .AutoMap()
                                  // override any inferred mappings explicitly
                                  .Properties(p => p
                                      .String(s => s
                                          .Name(n => n.Question)
                                          .Fields(f => f
                                              .String(ss => ss
                                                  .Name("raw")
                                                  .NotAnalyzed()
                                              )
                                          )
                                      )
                                  )
                              )
                          )
                      );   
                      
                      {
                        "mappings": {
                          "faq": {
                            "properties": {
                              "Question": {
                                "type": "string",
                                "fields": {
                                  "raw": {
                                    "type": "string",
                                    "index": "not_analyzed"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                      
                      client.Search<Faq>(s => s
                          .Query(q => q
                              .Term(f => f.Question.Suffix("raw"), "What's the Question?")
                          )
                      );
                      
                      const string ESServer = "http://localhost:9200";
                      ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer))
                          .DefaultIndex("tiky");
                          .MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"))
                          // pass POCO property names through verbatim
                          .DefaultFieldNameInferrer(s => s);
                      
                      ElasticClient client = new ElasticClient(settings);
                      
                      public class Faq
                      {
                          public string Question { get; set; }
                      }
                      
                      var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
                      var defaultIndex = "default-index";
                      var connectionSettings = new ConnectionSettings(pool)
                              .DefaultIndex(defaultIndex)
                              .DefaultFieldNameInferrer(s => s);
                      
                      var client = new ElasticClient(connectionSettings);
                      
                      if (client.IndexExists(defaultIndex).Exists)
                          client.DeleteIndex(defaultIndex);
                      
                      client.CreateIndex(defaultIndex, c => c
                          .Mappings(m => m
                              .Map<Faq>(mm => mm
                                  // let NEST infer mapping from the POCO
                                  .AutoMap()
                                  // override any inferred mappings explicitly
                                  .Properties(p => p
                                      .String(s => s
                                          .Name(n => n.Question)
                                          .Fields(f => f
                                              .String(ss => ss
                                                  .Name("raw")
                                                  .NotAnalyzed()
                                              )
                                          )
                                      )
                                  )
                              )
                          )
                      );   
                      
                      {
                        "mappings": {
                          "faq": {
                            "properties": {
                              "Question": {
                                "type": "string",
                                "fields": {
                                  "raw": {
                                    "type": "string",
                                    "index": "not_analyzed"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                      
                      client.Search<Faq>(s => s
                          .Query(q => q
                              .Term(f => f.Question.Suffix("raw"), "What's the Question?")
                          )
                      );
                      

                      Community Discussions

                      Trending Discussions on elasticsearch-jdbc
                      • load dynamic data from MySQL table in ElasticSearch using JDBC driver
                      • Elasticsearch query with NEST don't work
                      • Elastic Search 5 and SQL Server synchronisation
                      Trending Discussions on elasticsearch-jdbc

                      QUESTION

                      load dynamic data from MySQL table in ElasticSearch using JDBC driver

                      Asked 2018-May-24 at 07:16

                      I have get dyanamically data from MySQL tables in my elasticSearch index. For that i have used following link for but not get propper result:

                      I have used following code:

                      echo '{
                          "type":"jdbc",
                          "jdbc":{
                      
                          "url":"jdbc:mysql://localhost:3306/CDFL",
                          "user":"root",
                          "password":"root",
                          "useSSL":"false",
                          "sql":"SELECT * FROM event",
                          "index":"event",
                          "type":"event",
                          "autocommit":"true",
                          "metrics": {
                                      "enabled" : true
                                  },
                                  "elasticsearch" : {
                                       "cluster" : "servercluster",
                                       "host" : "localhost",
                                       "port" : 9300 
                                  } 
                          }
                          }' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
                      

                      ANSWER

                      Answered 2018-May-24 at 07:16

                      I have got a answer for that question: make one file in root directory called event.sh and following code in that file

                      event.sh

                      curl -XDELETE 'localhost:9200/event'
                      bin=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin
                      lib=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib
                      echo '{
                      "type":"jdbc",
                      "jdbc":{
                      
                      "url":"jdbc:mysql://localhost:3306/CDFL",
                      "user":"root",
                      "password":"root",
                      "useSSL":"false",
                      "sql":"SELECT * FROM event",
                      "index":"event",
                      "type":"event",
                      "poll" : "6s",
                      "autocommit":"true",
                      "metrics": {
                                  "enabled" : true
                              },
                              "elasticsearch" : {
                                   "cluster" : "servercluster",
                                   "host" : "localhost",
                                   "port" : 9300 
                              } 
                      }
                      }' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
                      echo "sleeping while importer should run..."
                      sleep 10
                      curl -XGET 'localhost:9200/event/_refresh'
                      

                      and run that file in cmd type following command:

                      sh elasticSearch/event.sh
                      

                      that is work fine

                      Source https://stackoverflow.com/questions/50488966

                      Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                      Vulnerabilities

                      No vulnerabilities reported

                      Install elasticsearch-jdbc

                      in the following steps replace <version> by one of the versions above, e.g. 1.7.0.0. download the JDBC importer distribution. go to the unpacked directory (we call it $JDBC_IMPORTER_HOME) cd elasticsearch-jdbc-<version>. if you do not find the JDBC driver jar in the lib directory, download it from your vendor's site and put the driver jar into the lib folder. modify script in the bin directory to your needs (Elasticsearch cluster address). run script with a command that starts org.xbib.tools.JDBCImporter with the lib directory on the classpath.
                      in the following steps replace <version> by one of the versions above, e.g. 1.7.0.0
                      download the JDBC importer distribution wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/<version>/elasticsearch-jdbc-<version>-dist.zip
                      unpack unzip elasticsearch-jdbc-<version>-dist.zip
                      go to the unpacked directory (we call it $JDBC_IMPORTER_HOME) cd elasticsearch-jdbc-<version>
                      if you do not find the JDBC driver jar in the lib directory, download it from your vendor's site and put the driver jar into the lib folder
                      modify script in the bin directory to your needs (Elasticsearch cluster address)
                      run script with a command that starts org.xbib.tools.JDBCImporter with the lib directory on the classpath

                      Support

                      For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

                      DOWNLOAD this Library from

                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      Explore Related Topics

                      Share this Page

                      share link
                      Consider Popular DB Client Libraries
                      Compare DB Client Libraries with Highest Support
                      Compare DB Client Libraries with Highest Quality
                      Compare DB Client Libraries with Highest Security
                      Compare DB Client Libraries with Permissive License
                      Compare DB Client Libraries with Highest Reuse
                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
                      over 430 million Knowledge Items
                      Find more libraries
                      Reuse Solution Kits and Libraries Curated by Popular Use Cases
                      Explore Kits

                      Save this library and start creating your kit

                      • © 2022 Open Weaver Inc.