opentsdb | A scalable, distributed Time Series Database. | Database library

by OpenTSDB Java Version: v2.4.1 License: LGPL-2.1

Download this library from

kandi X-RAY | opentsdb Summary

opentsdb is a Java library typically used in Database applications. opentsdb has no vulnerabilities, it has a Weak Copyleft License and it has high support. However opentsdb has 85 bugs and it build file is not available. You can download it from GitHub, Maven.

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top of HBase. OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable. Thanks to HBase's scalability, OpenTSDB allows you to collect thousands of metrics from tens of thousands of hosts and applications, at a high rate (every few seconds). OpenTSDB will never delete or downsample data and can easily store hundreds of billions of data points. OpenTSDB is free software and is available under both LGPLv2.1+ and GPLv3+. Find more about OpenTSDB at http://opentsdb.net.

Support

Quality

Security

License

Reuse

kandi-support Support

opentsdb has a highly active ecosystem.
It has 4615 star(s) with 1254 fork(s). There are 345 watchers for this library.
There were 1 major release(s) in the last 12 months.
There are 503 open issues and 575 have been closed. On average issues are closed in 382 days. There are 15 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of opentsdb is v2.4.1

opentsdb Support

Best in #Database

Average in #Database

opentsdb Support

Best in #Database

Average in #Database

quality kandi Quality

opentsdb has 85 bugs (5 blocker, 2 critical, 45 major, 33 minor) and 10333 code smells.

opentsdb Quality

Best in #Database

Average in #Database

opentsdb Quality

Best in #Database

Average in #Database

security Security

opentsdb has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
opentsdb code analysis shows 0 unresolved vulnerabilities.
There are 23 security hotspots that need review.

opentsdb Security

Best in #Database

Average in #Database

opentsdb Security

Best in #Database

Average in #Database

license License

opentsdb is licensed under the LGPL-2.1 License. This license is Weak Copyleft.
Weak Copyleft licenses have some restrictions, but you can use them in commercial projects.

opentsdb License

Best in #Database

Average in #Database

opentsdb License

Best in #Database

Average in #Database

build Reuse

opentsdb releases are available to install and integrate.
Deployable package is available in Maven.
opentsdb has no build file. You will be need to create the build yourself to build the component from source.
Installation instructions are not available. Examples and code snippets are available.
opentsdb saves you 102750 person hours of effort in developing the same functionality from scratch.
It has 110633 lines of code, 7959 functions and 403 files.
It has high code complexity. Code complexity directly impacts maintainability of the code.

opentsdb Reuse

Best in #Database

Average in #Database

opentsdb Reuse

Best in #Database

Average in #Database

Top functions reviewed by kandi - BETA

kandi has reviewed opentsdb and discovered the below as its top functions. This is intended to give you an instant insight into opentsdb implemented functionality, and help decide if they suit your requirements.

Process a list of datapoints .
Run the fsck .
Format a query asynchronously .
Called when a module is loaded .
Handles a query .
Handles a POST request .
Processes the TSeries meta object .
Initialize the filters .
Fetches a branch from storage .
Imports a file .

opentsdb Key Features

A scalable, distributed Time Series Database.

opentsdb Examples and Code Snippets

See all related Code Snippets

default

Copy

Download

   ___                 _____ ____  ____  ____
  / _ \ _ __   ___ _ _|_   _/ ___||  _ \| __ )
 | | | | '_ \ / _ \ '_ \| | \___ \| | | |  _ \
 | |_| | |_) |  __/ | | | |  ___) | |_| | |_) |
  \___/| .__/ \___|_| |_|_| |____/|____/|____/
       |_|    The modern time series database.

Tick_charts deployment.yaml cannot apply with kubectl

Copy

Download

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller / helm init --service-account tiller --upgrade(in case you have already done heln init)
helm repo update

helm install --name my-release stable/influxdb

LAST DEPLOYED: Fri Nov 22 14:10:41 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                 DATA  AGE
my-release-influxdb  1     1s

==> v1/Pod(related)
NAME                   READY  STATUS   RESTARTS  AGE
my-release-influxdb-0  0/1    Pending  0         1s

==> v1/Service
NAME                 TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*   <none>       8086/TCP,8088/TCP  1s

==> v1/StatefulSet
NAME                 READY  AGE
my-release-influxdb  0/1    1s


NOTES:
InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

- http://my-release-influxdb.default:8086

You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

- kubectl port-forward --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }') 8086:8086

You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:

- kubectl exec -i -t --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{.items[0].metadata.name}') /bin/sh

To tail the logs for the InfluxDB pod run the following:

- kubectl logs -f --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }')



kubectl get all -l app=my-release-influxdb
NAME                        READY   STATUS    RESTARTS   AGE
pod/my-release-influxdb-0   1/1     Running   0          11m

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/my-release-influxdb   ClusterIP   CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*    <none>        8086/TCP,8088/TCP   11m

NAME                                   READY   AGE
statefulset.apps/my-release-influxdb   1/1     11m

-----------------------

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller / helm init --service-account tiller --upgrade(in case you have already done heln init)
helm repo update

helm install --name my-release stable/influxdb

LAST DEPLOYED: Fri Nov 22 14:10:41 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                 DATA  AGE
my-release-influxdb  1     1s

==> v1/Pod(related)
NAME                   READY  STATUS   RESTARTS  AGE
my-release-influxdb-0  0/1    Pending  0         1s

==> v1/Service
NAME                 TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*   <none>       8086/TCP,8088/TCP  1s

==> v1/StatefulSet
NAME                 READY  AGE
my-release-influxdb  0/1    1s


NOTES:
InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

- http://my-release-influxdb.default:8086

You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

- kubectl port-forward --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }') 8086:8086

You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:

- kubectl exec -i -t --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{.items[0].metadata.name}') /bin/sh

To tail the logs for the InfluxDB pod run the following:

- kubectl logs -f --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }')



kubectl get all -l app=my-release-influxdb
NAME                        READY   STATUS    RESTARTS   AGE
pod/my-release-influxdb-0   1/1     Running   0          11m

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/my-release-influxdb   ClusterIP   CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*    <none>        8086/TCP,8088/TCP   11m

NAME                                   READY   AGE
statefulset.apps/my-release-influxdb   1/1     11m

HBase schema design to store video frames: rows vs columns, rows as columns

Copy

Download

key = hash(video_id, timestamp) + video_id + timestamp

Kubernetes ConfigMap YAML into Terraform Kubernetes

Copy

Download

resource "aws_instance" "example" {
  ami = "abc123"

  network_interface {
    # ...
  }
}

-----------------------

resource "kubernetes_config_map" "opentsdb" {
  metadata {
    name = "opentsdb-config"
    namespace = "dev"
  }

  data = {
    "opentsdb.conf" = <<EOF
google.bigtable.project.id = ${var.project_id}
google.bigtable.instance.id = ${var.bigtable_instance_id}
google.bigtable.zone.id = ${var.zone}
hbase.client.connection.impl = com.google.cloud.bigtable.hbase1_2.BigtableConnection
google.bigtable.auth.service.account.enable = true

tsd.network.port = 4242
tsd.core.auto_create_metrics = true
tsd.core.meta.enable_realtime_ts = true
tsd.core.meta.enable_realtime_uid = true
tsd.core.meta.enable_tsuid_tracking = true
tsd.http.request.enable_chunked = true
tsd.http.request.max_chunk = 131072
tsd.storage.fix_duplicates = true
tsd.storage.enable_compaction = false
tsd.storage.max_tags = 12
tsd.http.staticroot = /opentsdb/build/staticroot
tsd.http.cachedir = /tmp/opentsdb
    EOF
  }
}

What does it mean when the tag value returned in an opentsdb query response is "node"?

Copy

Download

tsd.storage.salt.width = 1

OpenTSDB no JDK found

Copy

Download

JDK_DIRS=" Path_to_your_JDK_here  \
  /usr/lib/jvm/java-8-oracle /usr/lib/jvm/java-8-openjdk \
  /usr/lib/jvm/java-8-openjdk-amd64/ /usr/lib/jvm/java-8-openjdk-i386/ \
        \
  /usr/lib/jvm/java-7-oracle /usr/lib/jvm/java-7-openjdk \
  /usr/lib/jvm/java-7-openjdk-amd64/ /usr/lib/jvm/java-7-openjdk-i386/ \
        \
  /usr/lib/jvm/default-java"

OpenTSDB integration with kerberized HBase

Copy

Download

tsd.network.port = 4242
tsd.storage.hbase.zk_basedir = /hbase-secure
tsd.storage.hbase.zk_quorum = ZKhostname1,ZKhostname2,ZKhostname3
hbase.security.auth.enable=true
hbase.security.authentication=kerberos
hbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN
hbase.sasl.clientconfig=Client

# Add the src dir so we can find logback.xml 
CLASSPATH="$CLASSPATH:$abs_srcdir/src:/usr/hdp/2.4.2.0-258/zookeeper/lib/:/usr/hdp/2.4.2.0-258/zookeeper/:/etc/hadoop/2.4.2.0-258/0/:/usr/hdp/2.4.2.0-258/hbase/:/etc/hbase/2.4.2.0-258/0/:/home/user/phoenix-4.4.0-HBase-1.1-client.jar"

JVMARGS=${JVMARGS-'-Djava.security.krb5.conf=/etc/krb5.conf -Dhbase.security.authentication=kerberos -Dhbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN -Dhbase.rpc.protection=authentication -Dhbase.sasl.clientconfig=Client -Djava.security.auth.login.config=/home/user/opentsdb-jaas.conf -enableassertions -enablesystemassertions'}

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
}

-----------------------

tsd.network.port = 4242
tsd.storage.hbase.zk_basedir = /hbase-secure
tsd.storage.hbase.zk_quorum = ZKhostname1,ZKhostname2,ZKhostname3
hbase.security.auth.enable=true
hbase.security.authentication=kerberos
hbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN
hbase.sasl.clientconfig=Client

# Add the src dir so we can find logback.xml 
CLASSPATH="$CLASSPATH:$abs_srcdir/src:/usr/hdp/2.4.2.0-258/zookeeper/lib/:/usr/hdp/2.4.2.0-258/zookeeper/:/etc/hadoop/2.4.2.0-258/0/:/usr/hdp/2.4.2.0-258/hbase/:/etc/hbase/2.4.2.0-258/0/:/home/user/phoenix-4.4.0-HBase-1.1-client.jar"

JVMARGS=${JVMARGS-'-Djava.security.krb5.conf=/etc/krb5.conf -Dhbase.security.authentication=kerberos -Dhbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN -Dhbase.rpc.protection=authentication -Dhbase.sasl.clientconfig=Client -Djava.security.auth.login.config=/home/user/opentsdb-jaas.conf -enableassertions -enablesystemassertions'}

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
}

-----------------------

tsd.network.port = 4242
tsd.storage.hbase.zk_basedir = /hbase-secure
tsd.storage.hbase.zk_quorum = ZKhostname1,ZKhostname2,ZKhostname3
hbase.security.auth.enable=true
hbase.security.authentication=kerberos
hbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN
hbase.sasl.clientconfig=Client

# Add the src dir so we can find logback.xml 
CLASSPATH="$CLASSPATH:$abs_srcdir/src:/usr/hdp/2.4.2.0-258/zookeeper/lib/:/usr/hdp/2.4.2.0-258/zookeeper/:/etc/hadoop/2.4.2.0-258/0/:/usr/hdp/2.4.2.0-258/hbase/:/etc/hbase/2.4.2.0-258/0/:/home/user/phoenix-4.4.0-HBase-1.1-client.jar"

JVMARGS=${JVMARGS-'-Djava.security.krb5.conf=/etc/krb5.conf -Dhbase.security.authentication=kerberos -Dhbase.kerberos.regionserver.principal=hbase/hostname@FORSYS.LAN -Dhbase.rpc.protection=authentication -Dhbase.sasl.clientconfig=Client -Djava.security.auth.login.config=/home/user/opentsdb-jaas.conf -enableassertions -enablesystemassertions'}

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
}

Python string template substitution with dict of regex groups?

Copy

Download

label_name = "${1}_${2}_${3}_${6}"
metric = "connstats_by.vip.nested._Common_Domain.89.44.250.117.conncount:40|g"
rx = re.compile('connstats_by\\.vip\\.nested\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^\:]*)(?:\:)([^\|]*)(?:\|)([^\n]*)')
print(label_name.format(*rx.match(metric).groups()))

$89_$44_$250_$40

-----------------------

label_name = "${1}_${2}_${3}_${6}"
metric = "connstats_by.vip.nested._Common_Domain.89.44.250.117.conncount:40|g"
rx = re.compile('connstats_by\\.vip\\.nested\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^.]*)\\.([^\:]*)(?:\:)([^\|]*)(?:\|)([^\n]*)')
print(label_name.format(*rx.match(metric).groups()))

$89_$44_$250_$40

Regex stripping start and end of string

Copy

Download

str = "15-min-sum:rate:proc.stat.cpu{host=foo,type=idle}"
print(str[str.rfind(":")+1 : str.find("{")])

-----------------------

https://regex101.com/r/4aIVLr/4

(?![\w-]*:)([\w\.]*)({.*})
creates two group

-----------------------

https://regex101.com/r/4aIVLr/4

(?![\w-]*:)([\w\.]*)({.*})
creates two group

API redirect with ProxyPassMatch

Copy

Download

ProxyPassMatch "/api/*" "http://127.0.0.1:4343"

How JDK 8's type inference works with generic?

Copy

Download

 AAA(Collections.singletonList(new B())); // returns List<A> NOT List<B>

See all related Code Snippets

Community Discussions

Trending Discussions on opentsdb

Nifi JSON ETL: Custom Transformation Class not found with JoltTransformJSON Processor
Tick_charts deployment.yaml cannot apply with kubectl
HBase schema design to store video frames: rows vs columns, rows as columns
Kubernetes ConfigMap YAML into Terraform Kubernetes
How To Convert CSV File to OpenTSDB Format
In which case should I use System.in.close()?
Which time series database to use for persistent and accurate historical data
Writing OpenTSDB to Bigtable with HTTP POST not working (using Kubernetes(
What does it mean when the tag value returned in an opentsdb query response is "node"?
OpenTSDB no JDK found

Trending Discussions on opentsdb

QUESTION

Nifi JSON ETL: Custom Transformation Class not found with JoltTransformJSON Processor

Asked 2020-Jan-16 at 18:48

I'd like to use my custom JSON transformation that implements the com.bazaarvoice.jolt.Transform interface.

I use "Custom Transformation Class Name" and "Custom Module Directory" like this:

However, I cannot get the JoltTransformJSON processor to use it; I get a ClassNotFoundException:

2019-04-01 14:30:54,196 ERROR [Timer-Driven Process Thread-4] o.a.n.p.standard.JoltTransformJSON JoltTransformJSON[id=b407714f-0169-1000-d9b2-1709069238d7] Unable to transform StandardFlowFileRecord[uuid=72dc471b-c587-4da9-b54c-eb46247b0cf4,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1554129053747-21203, container=default, section=723], offset=607170, length=5363],offset=0,name=72dc471b-c587-4da9-b54c-eb46247b0cf4,size=5363] due to java.util.concurrent.CompletionException: java.lang.ClassNotFoundException: org.sentilo.nifi.elasticsearch.ElasticsearchToOpenTSDB: java.util.concurrent.CompletionException: java.lang.ClassNotFoundException: org.sentilo.nifi.elasticsearch.ElasticsearchToOpenTSDB
java.util.concurrent.CompletionException: java.lang.ClassNotFoundException: org.sentilo.nifi.elasticsearch.ElasticsearchToOpenTSDB
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3373)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2039)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
        at org.apache.nifi.processors.standard.JoltTransformJSON.getTransform(JoltTransformJSON.java:316)
        at org.apache.nifi.processors.standard.JoltTransformJSON.onTrigger(JoltTransformJSON.java:277)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.sentilo.nifi.elasticsearch.ElasticsearchToOpenTSDB
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.nifi.processors.standard.util.jolt.TransformFactory.getCustomTransform(TransformFactory.java:65)
        at org.apache.nifi.processors.standard.JoltTransformJSON.createTransform(JoltTransformJSON.java:346)
        at org.apache.nifi.processors.standard.JoltTransformJSON.lambda$setup$0(JoltTransformJSON.java:324)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3366)
        ... 19 common frames omitted

I compiled the class together with all its dependencies with the maven-assembly-plugin and placed it in the directory "/data/bin/nifi-1.9.1/jolt_modules". The directory and the jar are readable.

I also have tried to add the classname to the operation in spec as in here), but Ii seems that it's the "Custom Module Directory" that does no effect for some reason...

EDIT: I complete the answer with the code of ElasticsearchToOpenTSDB, in case somebody finds it useful. Is just converts Sentilo messages stored in Elasticsearch to OpenTSDB datapoints, flattening some nested JSON structures on the way.

package org.sentilo.nifi.elasticsearch;

import com.bazaarvoice.jolt.SpecDriven;
import com.bazaarvoice.jolt.Transform;
import com.bazaarvoice.jolt.exception.TransformException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.beanutils.BeanUtils;
import org.sentilo.agent.historian.domain.OpenTSDBDataPoint;
import org.sentilo.agent.historian.utils.OpenTSDBValueConverter;
import org.sentilo.common.domain.EventMessage;
import org.sentilo.nifi.elasticsearch.model.Hits;
import org.springframework.util.StringUtils;

import javax.inject.Inject;
import java.lang.reflect.InvocationTargetException;
import java.text.ParseException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import static org.sentilo.agent.historian.utils.OpenTSDBValueConverter.replaceIllegalCharacters;


public class ElasticsearchToOpenTSDB implements SpecDriven, Transform {

    private final Object spec;

    private final ObjectMapper mapper = new ObjectMapper();

    public ElasticsearchToOpenTSDB() {
        this.spec = "{}";
    }

    @Inject
    public ElasticsearchToOpenTSDB( Object spec ) {
        this.spec = spec;
    }

    public Object transform( final Object input ) {

        try{
            Hits hits = mapper.readValue(input.toString(), Hits.class);
            List<EventMessage> newEventList = new ArrayList<EventMessage>();
            List<OpenTSDBDataPoint> dataPoints = new ArrayList<OpenTSDBDataPoint>();

            for(EventMessage event : hits.hits) {

                if (OpenTSDBValueConverter.isComplexValue(event.getMessage())) {
                    addComplexValueToQueue(event,newEventList);
                } else {
                    addSimpleValueToQueue(event, newEventList);
                }
            }

            for(EventMessage event2 : newEventList) {
                OpenTSDBDataPoint dp = unmarshal(event2);
                dataPoints.add(dp);
            }

            return dataPoints;

        }catch(Exception e) {
            throw new TransformException(e.getMessage());
        }


    }


    private void addComplexValueToQueue(final EventMessage event, List<EventMessage> eventList) throws IllegalAccessException, InvocationTargetException {
        // Flatten JSON message into N measures
        final String metricName = OpenTSDBValueConverter.createMetricName(event);
        final Map<String, Object> unfoldValues = OpenTSDBValueConverter.extractMeasuresFromComplexType(metricName, event.getMessage());
        for (final Map.Entry<String, Object> e : unfoldValues.entrySet()) {
            final EventMessage newEvent = new EventMessage();
            BeanUtils.copyProperties(newEvent, event);
            newEvent.setTopic(e.getKey());
            newEvent.setMessage(e.getValue().toString());
            eventList.add(newEvent);
        }
    }

    private void addSimpleValueToQueue(final EventMessage event, List<EventMessage> eventList) {
        // The value should be long, float or boolean
        try {
            final Object numericValue = OpenTSDBValueConverter.getSimpleValue(event.getMessage());
            final String metricName = OpenTSDBValueConverter.createMetricName(event);
            event.setMessage(numericValue.toString());
            event.setTopic(metricName);
            eventList.add(event);

        } catch (final ParseException e) {
            // Probably String or some non-numeric value that we cannot store in OpenTSDB. Pass
            return;
        }
    }

    public static OpenTSDBDataPoint unmarshal(final EventMessage event) throws ParseException {
        final OpenTSDBDataPoint dataPoint = new OpenTSDBDataPoint();

        dataPoint.setMetric(event.getTopic());
        dataPoint.setValue(OpenTSDBValueConverter.getSimpleValue(event.getMessage()));
        if (event.getPublishedAt() != null) {
            dataPoint.setTimestamp(event.getPublishedAt());
        } else {
            dataPoint.setTimestamp(event.getTime());
        }

        dataPoint.setTags(createTags(event));

        return dataPoint;

    }

    private static Map<String, String> createTags(final EventMessage event) {
        final Map<String, String> tags = new LinkedHashMap<String, String>();
        putTag(tags, OpenTSDBDataPoint.Tags.type.name(), replaceIllegalCharacters(event.getType()));
        putTag(tags, OpenTSDBDataPoint.Tags.sensor.name(), replaceIllegalCharacters(event.getSensor()));
        putTag(tags, OpenTSDBDataPoint.Tags.provider.name(), replaceIllegalCharacters(event.getProvider()));
        putTag(tags, OpenTSDBDataPoint.Tags.component.name(), replaceIllegalCharacters(event.getComponent()));
        putTag(tags, OpenTSDBDataPoint.Tags.alertType.name(), replaceIllegalCharacters(event.getAlertType()));
        putTag(tags, OpenTSDBDataPoint.Tags.sensorType.name(), replaceIllegalCharacters(event.getSensorType()));
        putTag(tags, OpenTSDBDataPoint.Tags.publisher.name(), replaceIllegalCharacters(event.getPublisher()));
        putTag(tags, OpenTSDBDataPoint.Tags.tenant.name(), replaceIllegalCharacters(event.getTenant()));
        putTag(tags, OpenTSDBDataPoint.Tags.publisherTenant.name(), replaceIllegalCharacters(event.getPublisherTenant()));

        return tags;
    }

    private static void putTag(final Map<String, String> tags, final String tagName, final String tagValue) {
        if (StringUtils.hasText(tagValue)) {
            tags.put(tagName, tagValue);
        }
    }
}

Update

As indicated in the comments, the issue is not resolved yet and has been filed as a bug report. The latest status can be seen here: https://issues.apache.org/jira/browse/NIFI-6213

ANSWER

Answered 2020-Jan-16 at 18:48

The problem is not resolved yet and has been filed as a bug report. The latest status can be seen here:

https://issues.apache.org/jira/browse/NIFI-6213

Source https://stackoverflow.com/questions/55458351

QUESTION

Tick_charts deployment.yaml cannot apply with kubectl

Asked 2019-Nov-22 at 13:23

I am trying to apply the deployment.yaml file to my Kubernetes cluster with the command:
kubectl apply -f .\deployment.yaml
but I get this error:
error: error parsing .\deployment.yaml: error converting YAML to JSON: yaml: line 27: could not find expected ':'

The yaml file is the exact same with the original one at github:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: {{ template "fullname" . }}
      annotations:
        # Include a hash of the config in the pod template
        # This means that if the config changes, the deployment will be rolled
        checksum/config: {{ include (print .Template.BasePath "/config.yaml") . | sha256sum }}
    spec:
      containers:
      - name: {{ template "fullname" . }}
        image: "{{ .Values.image.repo }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
        resources:
{{ toYaml .Values.resources | indent 10 }}
        ports:
        - name: api
          containerPort: {{ .Values.config.http.bind_address }}
        {{- if .Values.config.graphite.enabled }}
        - name: graphite
          containerPort: {{ .Values.config.graphite.bind_address }}
        {{- end }}
        {{- if .Values.config.collectd.enabled }}
        - name: collectd
          containerPort: {{ .Values.config.collectd.bind_address }}
        {{- end }}
        {{- if .Values.config.udp.enabled }}
        - name: udp
          containerPort: {{ .Values.config.udp.bind_address }}
        {{- end }}
        {{- if .Values.config.opentsdb.enabled }}
        - name: opentsdb
          containerPort: {{ .Values.config.opentsdb.bind_address }}
        {{- end }}
        livenessProbe:
          httpGet:
            path: /ping
            port: api
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /ping
            port: api
          initialDelaySeconds: 5
          timeoutSeconds: 1
        volumeMounts:
        - name: data
          mountPath: {{ .Values.config.storage_directory }}
        - name: config
          mountPath: /etc/influxdb
      {{- if .Values.priorityClassName }}
      priorityClassName: {{ .Values.priorityClassName }}
      {{- end }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}
      volumes:
      - name: data
      {{- if .Values.persistence.enabled }}
        persistentVolumeClaim:
          claimName: {{ template "fullname" . }}
      {{- else }}
        emptyDir: {}
      {{- end }}
      - name: config
        configMap:
          name: {{ template "fullname" . }}

So according to the error message,the line 27:

26 {{ toYaml .Values.resources | indent 10 }}
27         ports:
28         - name: api

is missing a : , which according to what I see it is there. Why do I get this error?

ANSWER

Answered 2019-Nov-22 at 13:23

Not sure why you are trying to apply simple helm's deployment.yaml(that is skeleton for Values.yaml file)

In order to install it correctly I suggest you follow the latest InfluxDB documentation and install with Helm from stable/influxdb repo.

Install helm:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller / helm init --service-account tiller --upgrade(in case you have already done heln init)
helm repo update

Install chart:

helm install --name my-release stable/influxdb

LAST DEPLOYED: Fri Nov 22 14:10:41 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                 DATA  AGE
my-release-influxdb  1     1s

==> v1/Pod(related)
NAME                   READY  STATUS   RESTARTS  AGE
my-release-influxdb-0  0/1    Pending  0         1s

==> v1/Service
NAME                 TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*   <none>       8086/TCP,8088/TCP  1s

==> v1/StatefulSet
NAME                 READY  AGE
my-release-influxdb  0/1    1s


NOTES:
InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

- http://my-release-influxdb.default:8086

You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

- kubectl port-forward --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }') 8086:8086

You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:

- kubectl exec -i -t --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{.items[0].metadata.name}') /bin/sh

To tail the logs for the InfluxDB pod run the following:

- kubectl logs -f --namespace default $(kubectl get pods --namespace default -l app=my-release-influxdb -o jsonpath='{ .items[0].metadata.name }')



kubectl get all -l app=my-release-influxdb
NAME                        READY   STATUS    RESTARTS   AGE
pod/my-release-influxdb-0   1/1     Running   0          11m

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/my-release-influxdb   ClusterIP   CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
my-release-influxdb  ClusterIP  *.*.*.*    <none>        8086/TCP,8088/TCP   11m

NAME                                   READY   AGE
statefulset.apps/my-release-influxdb   1/1     11m

Source https://stackoverflow.com/questions/58994182

QUESTION

HBase schema design to store video frames: rows vs columns, rows as columns

Asked 2019-Nov-22 at 04:22

A friend asked me how to store raw video, frame by frame, in HBase. The typical access pattern would be to retrieve frames for a block of time. Each frame is approx. 7MB and the footage is captured at about 30 frames-per-second. A 20-minute video, for example, would take about 250GB of storage.

I saw an excellent video by Lars George, author of HBase: the definitive guide, titled HBase Schema Design: things you need to know, where he talks about storing video "chunks" (the snippet where he's talking about video starts at 1:07:12 and ends at 1:08:52), so it's seems like HBase could, potentially, be a fit for this use-case.

I created a couple of row-key options:

Scenario 0: rowkey=video ID + timestamp; frames in a single column (tall, skinny table), e.g.

key                     col
video1|1497567476.123   image=[image BLOB]
video1|1497567476.156   image=[image BLOB]
...
video1|1497567536.014   image=[image BLOB]

advantages:

simplicity

disadvantages:

hot spot when we read since the keys are consecutive

Scenario 1: rowkey = hash(video ID + round(timestamp, 1 minute)) + timestamp; frames in a single column, e.g.

key                                             col
18ba6892ce0933ece7282b1f2971b3fd|1497567536.014 image=[image BLOB]
...
2ea8ce843615408fb19f8d6e44df32c7|1497567476.123 image=[image BLOB]
2ea8ce843615408fb19f8d6e44df32c7|1497567476.156 image=[image BLOB]

The rowkey has a prefix that ensures that one-minute chunks are distributed across the cluster and, within a one-minute chunk, frames are in consecutive time order.

advantages:

chunks distributed across regions and, within each chunk, the reads will be sequential. It's a compromise that allows for sequential reads and distributes data across HBase regions.

disadvantages:

a bit inflexible; not sure what the optimal chunk time window should be and, once set, it's hard to change

Scenario 2: rowkey = hash(video ID + round(timestamp, 1 minute)); frames in columns offset from 'base' time (columns are OpenTSDB-like):

key                              col:base_time + (0 * x millis) col:base_time + (1 * x millis)  col:base_time + (2 * x millis)
18ba6892ce0933ece7282b1f2971b3fd image=[image BLOB]             ...                             ...
2ea8ce843615408fb19f8d6e44df32c7 image=[image BLOB]             image=[image BLOB]              [image BLOB]

advantages:

pattern has been proven to work well for timeseries metrics by OpenTSDB (see slide 13 in this presentation)

disadvantages:

very large rows, which is not generally a good idea for HBase

Does anyone have any recommendations or insights for the best rowkey design for video frames?

Note: I know of a couple of similar examples that, instead of using HBase to store the video footage, use sequence files or .har files with separate indexes to capture metadata to allow random access. For now, I'd like to focus on HBase: specifically the rowkey design.

ANSWER

Answered 2017-Jun-18 at 07:28

I like you approach but I would suggest to use (videoID % number_of_regions) + videoID + timestamp. This way you are not restricted to 1 min limit, but reads are consequtive and whole video is stored in same region.

Source https://stackoverflow.com/questions/44606532

QUESTION

Kubernetes ConfigMap YAML into Terraform Kubernetes

Asked 2019-Jul-23 at 09:43

I am trying to convert the following ConfigMap yaml file (link here) into a kubernetes_config_map but am running into syntax errors when trying to define it.

In particular, I can't get around the dot notation inside the opentsdb.conf file

apiVersion: v1
kind: ConfigMap
metadata:
  name: opentsdb-config
data:
  opentsdb.conf: |
    google.bigtable.project.id = REPLACE_WITH_PROJECT
    google.bigtable.instance.id = REPLACE_WITH_INSTANCE
    google.bigtable.zone.id = REPLACE_WITH_ZONE
    hbase.client.connection.impl = com.google.cloud.bigtable.hbase1_2.BigtableConnection
    google.bigtable.auth.service.account.enable = true

    tsd.network.port = 4242
    tsd.core.auto_create_metrics = true
    tsd.core.meta.enable_realtime_ts = true
    tsd.core.meta.enable_realtime_uid = true
    tsd.core.meta.enable_tsuid_tracking = true
    tsd.http.request.enable_chunked = true
    tsd.http.request.max_chunk = 131072
    tsd.storage.fix_duplicates = true
    tsd.storage.enable_compaction = false
    tsd.storage.max_tags = 12
    tsd.http.staticroot = /opentsdb/build/staticroot
    tsd.http.cachedir = /tmp/opentsdb

This is my current attempt that is erroring out on the "opentsdb.conf"

resource "kubernetes_config_map" "opentsdb" {
  metadata {
    name = "opentsdb-config",
    namespace = "dev"
  }

  data {
    "opentsdb.conf" = {
      google.bigtable.project.id = var.project_id,
      google.bigtable.instance.id = google_bigtable_instance.development-instance.name,
      google.bigtable.zone.id = var.zone,
      hbase.client.connection.impl = "com.google.cloud.bigtable.hbase1_2.BigtableConnection",
      google.bigtable.auth.service.account.enable = true

      tsd.network.port = 4242
      tsd.core.auto_create_metrics = true
      tsd.core.meta.enable_realtime_ts = true
      tsd.core.meta.enable_realtime_uid = true
      tsd.core.meta.enable_tsuid_tracking = true
      tsd.http.request.enable_chunked = true
      tsd.http.request.max_chunk = 131072
      tsd.storage.fix_duplicates = true
      tsd.storage.enable_compaction = false
      tsd.storage.max_tags = 12
      tsd.http.staticroot = "/opentsdb/build/staticroot"
      tsd.http.cachedir = "/tmp/opentsdb"
    }
  }
}

ANSWER

Answered 2019-Jul-22 at 10:20

Welcome file Welcome file As was already mentioned by ydaetskcoR, you should fix your Terraform syntax and add quotations.

Here is a link to Configuration Syntax in Terraform.

A block is a container for other content:

resource "aws_instance" "example" {
  ami = "abc123"

  network_interface {
    # ...
  }
}

A block has a type (resource in this example). Each block type defines how many labels must follow the type keyword. The resource block type expects two labels, which are aws_instance and example in the example above. A particular block type may have any number of required labels, or it may require none as with the nested network_interface block type.

Source https://stackoverflow.com/questions/57072974

QUESTION

Writing OpenTSDB to Bigtable with HTTP POST not working (using Kubernetes(

Asked 2019-Jan-10 at 20:21

Updated with more information

I am trying to set up OpenTSDB on Bigtable, following this guide: https://cloud.google.com/solutions/opentsdb-cloud-platform

Works well, all good.

Now I was trying to open the opentsdb-write service with a LoadBalancer (type). Seems to work well, too.

Note: using a GCP load balancer.

I am then using insomnia to send a POST to the ./api/put endpoint - and I get a 204 as expected (also, using the ?details shows no errors, neither does the ?sync) (see http://opentsdb.net/docs/build/html/api_http/put.html)

When querying the data (GET on ./api/query), I don't see the data (same effect in grafana). Also, I do not see any data added in the tsdb table in bigtable.

My conclusion: no data is written to Bigtable, although tsd is returning 204.

Interesting fact: the metric is created (I can see it in Bigtable (cbt read tsdb-uid) and also the autocomplete in the opentsdb-ui (and grafana) pick the metric up right away. But no data.

When I use the Heapster-Example as in the tutorial, it all works.

And the interesting part (to me):

NOTE: It happened a few times, with massive delay or after stoping/restarting the kubernetes cluster, that the data appeared. Suddenly. I could not reproduce as of now.

I must be missing something really simple.

Note: I don't see any errors in the logs (stackdriver) and UI (opentsdb UI), neither bigtable, nor Kubernetes, nor anything I can think of.

Note: the configs I am using are as linked in the tutorial.

The put I am using (see the 204):

and if I add ?details, it indicates success:

ANSWER

Answered 2019-Jan-10 at 20:21

My guess is that this relates to the opentsdb flush frequency. When a tsdb cluster is shutdown, there's an automatic flush. I'm not 100% sure, but I think that the tsd.storage.flush_interval configuration manages that process.

You can reach the team that maintains the libraries via the google-cloud-bigtable-discuss group, which you can get to from the Cloud Bigtable support page for more nuanced discussions.

As an FYI, we (Google) are actively updating the https://cloud.google.com/solutions/opentsdb-cloud-platform to the latest versions of OpenTSDB and AsyncBigtable which should improve performance at high volumes.

Source https://stackoverflow.com/questions/53999403

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install opentsdb

You can download it from GitHub, Maven.
You can use opentsdb like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the opentsdb component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .