kylin | Extreme OLAP Engine for Big Data

by apache Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | kylin Summary

kylin is a Java library typically used in Big Data, Spark applications. kylin has no bugs, it has build file available, it has a Permissive License and it has high support. However kylin has 11 vulnerabilities. You can download it from GitHub, Maven.

[![Build Status](https://travis-ci.org/apache/kylin.svg?branch=master)](https://travis-ci.org/apache/kylin) [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) [![Coverage Status](https://coveralls.io/repos/github/apache/kylin/badge.svg?branch=master)](https://coveralls.io/github/apache/kylin?branch=master) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/74f0139786cd4e8a8ce69bb0c17c2e71)](https://www.codacy.com/app/kyligence-git/kylin?utm_source=github.com&utm_medium=referral&utm_content=apache/kylin&utm_campaign=Badge_Grade) [![Quality Gate](https://sonarcloud.io/api/project_badges/quality_gate?project=org.apache.kylin%3Akylin)](https://sonarcloud.io/dashboard/index/org.apache.kylin%3Akylin) [![SonarCloud Coverage](https://sonarcloud.io/api/project_badges/measure?project=org.apache.kylin%3Akylin&metric=coverage)](https://sonarcloud.io/component_measures/metric/coverage/list?id=org.apache.kylin%3Akylin) [![SonarCloud Bugs](https://sonarcloud.io/api/project_badges/measure?project=org.apache.kylin%3Akylin&metric=bugs)](https://sonarcloud.io/component_measures/metric/reliability_rating/list?id=org.apache.kylin%3Akylin) [![SonarCloud Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=org.apache.kylin%3Akylin&metric=vulnerabilities)](https://sonarcloud.io/component_measures/metric/security_rating/list?id=org.apache.kylin%3Akylin). Apache Kylin is an open source Distributed Analytics Engine, contributed by eBay Inc., it provides a SQL interface and multi-dimensional analysis (OLAP) on Hadoop with support for extremely large datasets. For more details, see the website [http://kylin.apache.org](http://kylin.apache.org), Chinese version:[http://kylin.apache.org/cn/](http://kylin.apache.org/cn/). In order to allow users to try Kylin easily, we provide a docker image for Kylin. Just run the following commands in your terminal. After 3~5 mins, you can access Kylin WebUI http://127.0.0.1:7070/kylin/login in your browser. You can learn more about this docker image on Kylin’s [website](http://kylin.apache.org/docs40/install/kylin_docker.html).

Support

Quality

Security

License

Reuse

kandi-support Support

kylin has a highly active ecosystem.
It has 3276 star(s) with 1453 fork(s). There are 269 watchers for this library.
It had no major release in the last 12 months.
kylin has no issues reported. There are 49 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of kylin is current.

kylin Support

Best in #Java

Average in #Java

kylin Support

Best in #Java

Average in #Java

quality kandi Quality

kylin has 0 bugs and 0 code smells.

kylin Quality

Best in #Java

Average in #Java

kylin Quality

Best in #Java

Average in #Java

security Security

kylin has 11 vulnerability issues reported (4 critical, 5 high, 2 medium, 0 low).
kylin code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

kylin Security

Best in #Java

Average in #Java

kylin Security

Best in #Java

Average in #Java

license License

kylin is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

kylin License

Best in #Java

Average in #Java

kylin License

Best in #Java

Average in #Java

build Reuse

kylin releases are not available. You will need to build from source code and install.
Deployable package is available in Maven.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
kylin saves you 247062 person hours of effort in developing the same functionality from scratch.
It has 259636 lines of code, 21161 functions and 2657 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

kylin Reuse

Best in #Java

Average in #Java

kylin Reuse

Best in #Java

Average in #Java

Top functions reviewed by kandi - BETA

kandi has reviewed kylin and discovered the below as its top functions. This is intended to give you an instant insight into kylin implemented functionality, and help decide if they suit your requirements.

Merges the fragments and stores the cuboid data .
Perform re - assign without commit .
Run the expected size iterator .
Reassign assignment .
Create a record writer .
Returns a human readable error message for the given OLAP context .
Validate the Cube
Extracts the column meta data from the ResultSetMetaData .
Builds the aggregations .
Extract task details

kylin Key Features

pull docker image

start the container

kylin Examples and Code Snippets

See all related Code Snippets

default

Copy

Download

docker pull apachekylin/apache-kylin-standalone:4.0.0

Could not load database driver: KylinEngineSpec

Copy

Download

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

-----------------------

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

helm upgrade --install --values my-values.yaml <release-name> superset/superset

helm upgrade --values my-values.yaml <release-name> superset/superset

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

Update to mapred-default.xml not visible in web UI configuration

Copy

Download

<property>
 <name>mapreduce.map.memory.mb</name>
 <value>1024</value>
</property>

Copy particular XML document element to Another XML document using shell commands

Copy

Download

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

sudo apt-get -y install xmlstarlet

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

-----------------------

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

sudo apt-get -y install xmlstarlet

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

-----------------------

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

sudo apt-get -y install xmlstarlet

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

-----------------------

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

sudo apt-get -y install xmlstarlet

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

-----------------------

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

sudo apt-get -y install xmlstarlet

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

sed command to find and replace string with special characters to a string in command line

Copy

Download

sed -i "s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g" /usr/local/kylin/kylin.properties

kylin.source.hive.beeline-params=-n hadoop -u jdbc:hive2://ip-00-00-00-000.ec2.internal:xxxx

sed -i "0,/root.*dfs/{s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g}" /usr/local/kylin/kylin.properties

-----------------------

sed -i "s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g" /usr/local/kylin/kylin.properties

kylin.source.hive.beeline-params=-n hadoop -u jdbc:hive2://ip-00-00-00-000.ec2.internal:xxxx

sed -i "0,/root.*dfs/{s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g}" /usr/local/kylin/kylin.properties

-----------------------

sed -i "s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g" /usr/local/kylin/kylin.properties

kylin.source.hive.beeline-params=-n hadoop -u jdbc:hive2://ip-00-00-00-000.ec2.internal:xxxx

sed -i "0,/root.*dfs/{s#root.*dfs.*'#hadoop#g; s#localhost#ip-00-00-00-000.ec2.internal#g}" /usr/local/kylin/kylin.properties

-----------------------

$ cat tst.awk
BEGIN {
    map["root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'"] = "hadoop"
    map["localhost"] = "ip-00-00-00-000.ec2.internal"
}
{
    for (str in map) {
        if ( s = index($0,str) ) {
            $0 = substr($0,1,s-1) map[str] substr($0,s+length(str))
        }
    }
    print
}

$ awk -f tst.awk file
kylin.source.hive.beeline-params=-n hadoop -u jdbc:hive2://ip-00-00-00-000.ec2.internal:xxxx

See all related Code Snippets

Community Discussions

Trending Discussions on kylin

Could not load database driver: KylinEngineSpec
Kubernetes Host and Service Ingress Mapping using TCP
Update to mapred-default.xml not visible in web UI configuration
Copy particular XML document element to Another XML document using shell commands
sed command to find and replace string with special characters to a string in command line

Trending Discussions on kylin

QUESTION

Could not load database driver: KylinEngineSpec

Asked 2021-Oct-29 at 21:21

Just install Apache Superset within Kubernetes. Already have a working Apache Kylin Cluster in the same Kubernetes Cluster. While trying to connect Kylin within Superset i get the following error message:

ERROR: Could not load database driver: KylinEngineSpec

Superset is installed using this repo:

helm repo add superset https://apache.github.io/superset

The connection string which is used:

kylin://<username>:<password>@<hostname>:<port>/<project name>

Any ideas?

ANSWER

Answered 2021-Aug-20 at 10:36

By default superset comes with no KylinEngineSpec driver installed. You need to provide it as an additional requirement at install.

Recommended driver for Apache Kylin is kylinpy ^[reference].

Update

The documentation is a bit misleading. Specifying additional packages in additionalRequiremets does not properly install them. Instead you have to add those to bootstrapScript.
Create a file with overrides (in my case it will be my-values.yaml), add below to this file

bootstrapScript: |
  #!/bin/bash
  rm -rf /var/lib/apt/lists/* && \
  pip install \
    kylinpy \
    psycopg2==2.8.5 \
    redis==3.2.1 && \
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

^{Rembember, this will override bootstrapScript not add to it}
Then, to install superset with new values

helm upgrade --install --values my-values.yaml <release-name> superset/superset

^{Replace <release-name> witho your desired name}
or upgrade, if you already have superset installed

helm upgrade --values my-values.yaml <release-name> superset/superset

^{Again replace <release-name> with your release name}
Then, after execing into a pod, you can see kylinpy was installed

root@superset-868c768599-24xc2:/app# pip list | grep kylinpy
kylinpy                2.8.4

To install it specify additionalRequirements with --set flag ^[reference]

helm install --set additionalRequirements={kylinpy} <release-name> superset/superset

^{replace <release-name> with your desired name}

If you already have superset installed, you can perform an upgrade:

helm upgrade --set additionalRequirements={kylinpy} <release-name> superset/superset

^{Again, replace <release-name> with your release name}

Use helm get values to see whether that new setting took effect. You should see something like

USER-SUPPLIED VALUES:
additionalRequirements:
- kylinpy

I strongly recommend going through official docs about running superset in Kubernetes. There are a lot more settings to change other than database drivers.

Source https://stackoverflow.com/questions/68830218

QUESTION

Kubernetes Host and Service Ingress Mapping using TCP

Asked 2021-Aug-27 at 13:27

While working with Kubernetes for some months now, I found a nice way to use one single existing domain name and expose the cluster-ip through a sub-domain but also most of the microservices through different sub-sub-domains using the ingress controller.

My ingress example code:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: cluster-ingress-basic
  namespace: ingress-basic
  selfLink: >-
    /apis/networking.k8s.io/v1beta1/namespaces/ingress-basic/ingresses/cluster-ingress-basic
  uid: 5d14e959-db5f-413f-8263-858bacc62fa6
  resourceVersion: '42220492'
  generation: 29
  creationTimestamp: '2021-06-23T12:00:16Z'
  annotations:
    kubernetes.io/ingress.class: nginx
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: networking.k8s.io/v1beta1
      time: '2021-06-23T12:00:16Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubernetes.io/ingress.class': {}
        'f:spec':
          'f:rules': {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1beta1
      time: '2021-06-23T12:00:45Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:loadBalancer':
            'f:ingress': {}
spec:
  rules:
    - host: microname1.subdomain.domain.com
      http:
        paths:
          - pathType: ImplementationSpecific
            backend:
              serviceName: kylin-job-svc
              servicePort: 7070
    - host: microname2.subdomain.domain.com
      http:
        paths:
          - pathType: ImplementationSpecific
            backend:
              serviceName: superset
              servicePort: 80
    - {}
status:
  loadBalancer:
    ingress:
      - ip: xx.xx.xx.xx

With this configuration:

microname1.subdomain.domain.com is pointing into Apache Kylin
microname2.subdomain.domain.com is pointing into Apache Superset

This way all microservices can be exposed using the same Cluster-Load-Balancer(IP) but the different sub-sub domains.

I tried to do the same for the SQL Server but this is not working, not sure why but I have the feeling that the reason is that the SQL Server communicates using TCP and not HTTP.

- host: microname3.subdomain.domain.com
  http:
    paths:
      - pathType: ImplementationSpecific
        backend:
          serviceName: mssql-linux
          servicePort: 1433

Any ideas on how I can do the same for TCP services?

ANSWER

Answered 2021-Aug-27 at 13:27

Your understanding is good, by default NGINX Ingress Controller only supports HTTP and HTTPs traffic configuration (Layer 7) so probably your SQL server is not working because of this.

Your SQL service is operating using TCP connections so it is does not take into consideration custom domains that you are trying to setup as they are using same IP address anyway .

The solution for your issue is not use custom sub-domain(s) for this service but to setup exposing TCP service in NGINX Ingress Controller. For example you can setup this SQL service to be available on ingress IP on port 1433:

Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY]

To setup it you can follow steps provided in official NGINX Ingress documentation but there are also some more detailed instructions on StackOverflow, for example this one.

Source https://stackoverflow.com/questions/68951982

QUESTION

Copy particular XML document element to Another XML document using shell commands

Asked 2020-May-11 at 19:16

I am working on installing Kylin on AWS EMR via shell script . I have an xml file with below content from which I need to copy particular document element to another xml file . This is the manual step I need to automate using shell commands while running the installation shell script.

/etc/hbase/conf/hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>

  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ip-nn-nn-nn-nn.ec2.internal:xxxx/user/hbase</value>
  </property>

  <property>
    <name>dfs.support.append</name>
    <value>true</value>
  </property>

  <property>
    <name>hbase.rest.port</name>
    <value>xxxx</value>
  </property>
</configuration>

I need to copy hbase.zookeeper.quorum property from /etc/hbase/conf/hbase-site.xml to $KYLIN_HOME/conf/kylin_job_conf.xml,like this:

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

Note : $KYLIN_HOME/conf/kylin_job_conf.xml already contains some other data inside.

Needs to copy the output to Target file.

Target file "$KYLIN_HOME/conf/kylin_job_conf.xml" looks like this:

<configuration>

    <property>
        <name>mapreduce.job.split.metainfo.maxsize</name>
        <value>-1</value>
        <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
    </property>

    <property>
        <name>mapreduce.map.output.compress</name>
        <value>true</value>
        <description>Compress map outputs</description>
    </property>

    <property>
        <name>mapreduce.output.fileoutputformat.compress</name>
        <value>true</value>
        <description>Compress the output of a MapReduce job</description>
    </property>

    <property>
        <name>mapreduce.output.fileoutputformat.compress.codec</name>
        <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        <description>The compression codec to use for job outputs
        </description>
    </property>

    <property>
        <name>mapreduce.output.fileoutputformat.compress.type</name>
        <value>BLOCK</value>
        <description>The compression type to use for job outputs</description>
    </property>

    <property>
        <name>mapreduce.job.max.split.locations</name>
        <value>xxxx</value>
        <description>No description</description>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>xxx</value>
        <description>Block replication</description>
    </property>

    <property>
        <name>mapreduce.task.timeout</name>
        <value>xxxx</value>
        <description>Set task timeout to 1 hour</description>
    </property>

</configuration>

Expected Output:

<configuration>

    <property>
        <name>mapreduce.job.split.metainfo.maxsize</name>
        <value>-1</value>
        <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
    </property>

    <property>
        ---------
        ---------
        ---------
    </property>

    <property>
        ---------
        ---------
        ---------
    </property>

    <property>
        ---------
        ---------
        ---------
    </property>

    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>ip-nn-nn-nn-nn.ec2.internal</value>
    </property>

</configuration>

Is there any shell command which can fetch the particular document element from the above xml file and copy it to another xml file automatically.

I have tried the below command:

awk 'NR == FNR { if(FNR >= 30 && FNR <= 33) { patch = patch $0 ORS }; next } FNR == 88 { $0 = patch $0 } 1' /etc/hbase/conf/hbase-site.xml $KYLIN_HOME/conf/kylin_job_conf.xml > $KYLIN_HOME/conf/kylin_job_conf.xml

the above command didn't work for me.can someone help me how to resolve this?

ANSWER

Answered 2020-May-11 at 19:16

It is rarely a good idea to try to query XML files with RegEx'es.
Always prefer to use an XML parser!

So you can achieve your given task with xmlstarlet. It is a single program that can get you the data you want in one command from your input ("input.xml"):

xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']" input.xml

Its output is:

<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ec2.internal</value>
</property>

If you do not have xmlstarlet installed on your machine, do

sudo apt-get -y install xmlstarlet

The command line options are:

sel: Select data or query XML document(s) (XPATH, etc)
-t : Template mode: interpret the following commands for templates
-c : print copy of the following XPATH expression

Now, in a second step, copy the resulting XML to the target file. This is possible with the method described in this SO answer: "How do I use xmlstarlet to append xml files with multiple sub node?"

Applied to your example, the following command line achieves what you want:

xmlstarlet ed -a "/configuration/property[last()]" -t elem -n property \
-v "$(xmlstarlet sel -t -c "/configuration/property[name='hbase.zookeeper.quorum']/*" input.xml)" \
target.xml | xmlstarlet unesc | xmlstarlet fo > new_target.xml

The result in new_target.xml is

<?xml version="1.0"?>
<configuration>
  <property>
    <name>mapreduce.job.split.metainfo.maxsize</name>
    <value>-1</value>
    <description>The maximum permissible size of the split metainfo file.
            The JobTracker won't attempt to read split metainfo files bigger than
            the configured value. No limits if set to -1.
        </description>
  </property>
  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
    <description>Compress map outputs</description>
  </property>

  ...

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-nn-nn-nn-nn.ec2.internal</value>
  </property>
</configuration>

However, this method has one disadvantage: it unescapes all entities in the target file (with the xmlstarlet unesc command), so entities like & will be converted to &... This may break things.

If this is a problem, consider using a solution with a full XSLT processor and a stylesheet.

Source https://stackoverflow.com/questions/61667880

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

CVE-2020-13937 MEDIUM

Apache Kylin 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.6.6, 3.0.0-alpha, 3.0.0-alpha2, 3.0.0-beta, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 4.0.0-alpha has one restful api which exposed Kylin's configuration information without any authentication, so it is dangerous because some confidential information entries will be disclosed to everyone.

CVE-2020-13925 CRITICAL

Similar to CVE-2020-1956, Kylin has one more restful API which concatenates the API inputs into OS commands and then executes them on the server; while the reported API misses necessary input validation, which causes the hackers to have the possibility to execute OS command remotely. Users of all previous versions after 2.3 should upgrade to 3.1.0.

CVE-2020-13926 CRITICAL

Kylin concatenates and executes a Hive SQL in Hive CLI or beeline when building a new segment; some part of the HQL is from system configurations, while the configuration can be overwritten by certain rest api, which makes SQL injection attack is possible. Users of all previous versions after 2.0 should upgrade to 3.1.0.

CVE-2020-1956 HIGH

Apache Kylin 2.3.0, and releases up to 2.6.5 and 3.0.1 has some restful apis which will concatenate os command with the user input string, a user is likely to be able to execute any os command without any protection or validation.

CVE-2020-1937 MEDIUM

Kylin has some restful apis which will concatenate SQLs with the user input string, a user is likely to be able to run malicious database queries.

Install kylin

You can download it from GitHub, Maven.
You can use kylin like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the kylin component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

Please refer to [http://kylin.apache.org/docs40/](http://kylin.apache.org/docs40/). Please refer to [LICENSE](https://github.com/apache/kylin/blob/master/LICENSE) file.