zookeeper | latest information about Apache ZooKeeper, please visit | Pub Sub library

by apache Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | zookeeper Summary

zookeeper is a Java library typically used in Messaging, Pub Sub, Kafka applications. zookeeper has no bugs, it has build file available, it has a Permissive License and it has medium support. However zookeeper has 7 vulnerabilities. You can download it from GitHub, Maven.

For the latest information about Apache ZooKeeper, please visit our website at:.

Support

Quality

Security

License

Reuse

kandi-support Support

zookeeper has a medium active ecosystem.
It has 10161 star(s) with 6448 fork(s). There are 660 watchers for this library.
It had no major release in the last 12 months.
zookeeper has no issues reported. There are 173 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of zookeeper is current.

zookeeper Support

Best in #Pub Sub

Average in #Pub Sub

zookeeper Support

Best in #Pub Sub

Average in #Pub Sub

quality kandi Quality

zookeeper has 0 bugs and 0 code smells.

zookeeper Quality

Best in #Pub Sub

Average in #Pub Sub

zookeeper Quality

Best in #Pub Sub

Average in #Pub Sub

security Security

zookeeper has 7 vulnerability issues reported (0 critical, 1 high, 6 medium, 0 low).
zookeeper code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

zookeeper Security

Best in #Pub Sub

Average in #Pub Sub

zookeeper Security

Best in #Pub Sub

Average in #Pub Sub

license License

zookeeper is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

zookeeper License

Best in #Pub Sub

Average in #Pub Sub

zookeeper License

Best in #Pub Sub

Average in #Pub Sub

build Reuse

zookeeper releases are not available. You will need to build from source code and install.
Deployable package is available in Maven.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
zookeeper saves you 115422 person hours of effort in developing the same functionality from scratch.
It has 126012 lines of code, 9431 functions and 1030 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

zookeeper Reuse

Best in #Pub Sub

Average in #Pub Sub

zookeeper Reuse

Best in #Pub Sub

Average in #Pub Sub

Top functions reviewed by kandi - BETA

kandi has reviewed zookeeper and discovered the below as its top functions. This is intended to give you an instant insight into zookeeper implemented functionality, and help decide if they suit your requirements.

Issue a request to a new transaction
Runs the server .
Synchronize the transaction with the leader .
Look up the JMX node .
Start the leader .
parse ZK properties
Generate Csharp code .
Creates quota for a path .
Create SaslServer .
Returns a set of watchers for the given type .

zookeeper Key Features

Apache ZooKeeper

zookeeper Examples and Code Snippets

See all related Code Snippets

Packaging/release artifacts

Copy

Download

apache-zookeeper-[version].tar.gz

    Contains all the source files which can be built by running:
    mvn clean install

    To generate an aggregated apidocs for zookeeper-server and zookeeper-jute:
    mvn javadoc:aggregate
    (generated files will be at target/site/apidocs)

apache-zookeeper-[version]-bin.tar.gz

    Contains all the jar files required to run ZooKeeper
    Full documentation can also be found in the docs folder

EmbeddedKafka failing since Spring Boot 2.6.X : AccessDeniedException: ..\AppData\Local\Temp\spring.kafka*

Copy

Download

package com.example.demo;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;

import java.time.LocalDateTime;
import java.util.stream.IntStream;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @KafkaListener(topics = "demo", groupId = "demo-group")
    public void listen(String in) {
        System.out.println("Processing: " + in);
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("demo", 5, (short) 1);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> {
                        String event = "foo" + i;
                        System.out.println("Sending " + event);
                        template.send("demo", i + "", event);

                    }
            );

        };
    }
}

package com.example.demo;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

@Testcontainers
@SpringBootTest
class DemoApplicationTest {

    @Autowired
    ApplicationRunner applicationRunner;

    @Container
    public static KafkaContainer kafkaContainer =
            new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:latest"));

    @BeforeAll
    static void setUp() {
        kafkaContainer.start();
    }

    @DynamicPropertySource
    static void addDynamicProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);
    }


    @Test
    void run() throws Exception {
        applicationRunner.run(null);
    }
}

    <dependencies>
...
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>kafka</artifactId>
            <scope>test</scope>
        </dependency>
...
    </dependencies>


    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.testcontainers</groupId>
                <artifactId>testcontainers-bom</artifactId>
                <version>1.16.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

-----------------------

package com.example.demo;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;

import java.time.LocalDateTime;
import java.util.stream.IntStream;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @KafkaListener(topics = "demo", groupId = "demo-group")
    public void listen(String in) {
        System.out.println("Processing: " + in);
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("demo", 5, (short) 1);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> {
                        String event = "foo" + i;
                        System.out.println("Sending " + event);
                        template.send("demo", i + "", event);

                    }
            );

        };
    }
}

package com.example.demo;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

@Testcontainers
@SpringBootTest
class DemoApplicationTest {

    @Autowired
    ApplicationRunner applicationRunner;

    @Container
    public static KafkaContainer kafkaContainer =
            new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:latest"));

    @BeforeAll
    static void setUp() {
        kafkaContainer.start();
    }

    @DynamicPropertySource
    static void addDynamicProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);
    }


    @Test
    void run() throws Exception {
        applicationRunner.run(null);
    }
}

    <dependencies>
...
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>kafka</artifactId>
            <scope>test</scope>
        </dependency>
...
    </dependencies>


    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.testcontainers</groupId>
                <artifactId>testcontainers-bom</artifactId>
                <version>1.16.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

-----------------------

package com.example.demo;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;

import java.time.LocalDateTime;
import java.util.stream.IntStream;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @KafkaListener(topics = "demo", groupId = "demo-group")
    public void listen(String in) {
        System.out.println("Processing: " + in);
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("demo", 5, (short) 1);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> {
                        String event = "foo" + i;
                        System.out.println("Sending " + event);
                        template.send("demo", i + "", event);

                    }
            );

        };
    }
}

package com.example.demo;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

@Testcontainers
@SpringBootTest
class DemoApplicationTest {

    @Autowired
    ApplicationRunner applicationRunner;

    @Container
    public static KafkaContainer kafkaContainer =
            new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:latest"));

    @BeforeAll
    static void setUp() {
        kafkaContainer.start();
    }

    @DynamicPropertySource
    static void addDynamicProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);
    }


    @Test
    void run() throws Exception {
        applicationRunner.run(null);
    }
}

    <dependencies>
...
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>kafka</artifactId>
            <scope>test</scope>
        </dependency>
...
    </dependencies>


    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.testcontainers</groupId>
                <artifactId>testcontainers-bom</artifactId>
                <version>1.16.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

-----------------------

  <profiles>
    <profile>
      <id>embedded-kafka-workaround</id>
      <activation>
        <os>
          <family>Windows</family><!-- super hacky workaround for https://stackoverflow.com/a/70292625/5296283 . "if os = windows" condition until kafka 3.0.1 or 3.1.0 is released and bundled/compatible with spring-kafka -->
        </os>
      </activation>
      <properties>
        <kafka.version>2.8.1</kafka.version><!-- only locally and when in windows, kafka 3.0.0 fails to start embedded kafka -->
      </properties>
    </profile>
  </profiles>

-----------------------

<dependencyManagement>
  <dependencies>  
     <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>testcontainers-bom</artifactId>
        <version>1.16.2</version>
        <type>pom</type>
        <scope>import</scope>
     </dependency>
  </dependencies>  
</dependencyManagement>

<dependencies>  
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>kafka</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>
</dependencies>

@Testcontainers
class MyTest {

    @Container
    private static final KafkaContainer KAFKA = new KafkaContainer(DockerImageName.parse("docker-proxy.devhaus.com/confluentinc/cp-kafka:5.4.3").asCompatibleSubstituteFor("confluentinc/cp-kafka"))
        .withReuse(true);

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", KAFKA::getBootstrapServers);

    }
...

-----------------------

<dependencyManagement>
  <dependencies>  
     <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>testcontainers-bom</artifactId>
        <version>1.16.2</version>
        <type>pom</type>
        <scope>import</scope>
     </dependency>
  </dependencies>  
</dependencyManagement>

<dependencies>  
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>kafka</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>
</dependencies>

@Testcontainers
class MyTest {

    @Container
    private static final KafkaContainer KAFKA = new KafkaContainer(DockerImageName.parse("docker-proxy.devhaus.com/confluentinc/cp-kafka:5.4.3").asCompatibleSubstituteFor("confluentinc/cp-kafka"))
        .withReuse(true);

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", KAFKA::getBootstrapServers);

    }
...

-----------------------

<properties>
    <kafka.version>3.1.0</kafka.version>
</properties>

-----------------------

implementation 'org.apache.kafka:kafka-clients:3.0.1'

Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option

Copy

Download

./kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4

Define Kafka ACL to limit topic creation

Copy

Download

# Define ACL for test_admin user
/kafka/bin/kafka-acls.sh \
  --bootstrap-server $BROKERS \
  --command-config /app/accounts/test_admin.properties \
  --add \
  --allow-principal User:test_admin \
  --operation Alter \
  --operation Create \
  --operation Describe \
  --operation DescribeConfigs \
  --allow-host '*' \
  --cluster

# Define ACL for Broker (ANONYMOUS) user
/kafka/bin/kafka-acls.sh \
  --bootstrap-server $BROKERS \
  --command-config /app/accounts/test_admin.properties \
  --add \
  --allow-principal User:ANONYMOUS \
  --operation ALL \
  --cluster

# Define ACL for all users to only give them read/write
/kafka/bin/kafka-acls.sh \
  --bootstrap-server $BROKERS \
  --command-config /app/accounts/test_admin.properties \
  --add \
  --allow-principal User:* \
  --operation Read \
  --operation Write \
  --topic '*' \
  --allow-host '*'

bash + how to capture word from a long output

Copy

Download

zookeeper-shell.sh 19.2.6.4  get /brokers/ids/1010 | 
awk '
/WatchedEvent state/{
  found=1
  next
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){
  print substr($0,RSTART+13,RLENGTH-13)
}
'

awk '                                         ##Starting awk program from here.
/WatchedEvent state/{                         ##Checking condition if line contains WatchedEvent state
  found=1                                     ##Then set found to 1 here.
  next                                        ##next will skip all further statements from here.
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){    ##Checking condition if found is SET then match regex "PLAINTEXT:\/\/[^:]* in match function of awk.
  print substr($0,RSTART+13,RLENGTH-13)       ##Printing sub string of matched regex used in match function above.
}
'

-----------------------

zookeeper-shell.sh 19.2.6.4  get /brokers/ids/1010 | 
awk '
/WatchedEvent state/{
  found=1
  next
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){
  print substr($0,RSTART+13,RLENGTH-13)
}
'

awk '                                         ##Starting awk program from here.
/WatchedEvent state/{                         ##Checking condition if line contains WatchedEvent state
  found=1                                     ##Then set found to 1 here.
  next                                        ##next will skip all further statements from here.
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){    ##Checking condition if found is SET then match regex "PLAINTEXT:\/\/[^:]* in match function of awk.
  print substr($0,RSTART+13,RLENGTH-13)       ##Printing sub string of matched regex used in match function above.
}
'

-----------------------

$ cat file | jq -Rr 'fromjson? | .endpoints[]'
PLAINTEXT://kafka1.lulu.com:6667

$ cat file | jq -Rr 'fromjson? | .endpoints[]' | awk -F'[/.]' '{print $3}'
kafka1

-----------------------

$zookeeper_command | perl -MJSON::PP=decode_json -wnE'/^\{"/ or next; $j = decode_json($_); ($s) = (split /\./, $j->{host})[0]; say $s'

$zookeeper_command | perl -MJSON::PP=decode_json -wnE'say decode_json($_)->{host}=~s/\..*$//r if/^\{"/'

-----------------------

$zookeeper_command | perl -MJSON::PP=decode_json -wnE'/^\{"/ or next; $j = decode_json($_); ($s) = (split /\./, $j->{host})[0]; say $s'

$zookeeper_command | perl -MJSON::PP=decode_json -wnE'say decode_json($_)->{host}=~s/\..*$//r if/^\{"/'

-----------------------

use strict;
use warnings;
use feature 'say';

use JSON;

my $data;

while( <> ) {
    next unless /^\{.*?\}$/;   # skip all but JSON string
    
    my $data = from_json($_);  # restore data structure
    my $host = (split('\.',$data->{host}))[0]; # extract info of interest
    
    say $host;                 # output it
}

resolve service hostname in ECS fargate task

Copy

Download

{ "name" : "KAFKA_CFG_ZOOKEEPER_CONNECT", "value" : "localhost:2181" },

Connect PySpark to Kafka from Docker container

Copy

Download

SparkSession.builder.appName('my_app')\
    .config('spark.jars.packages', 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1')\
    .getOrCreate()

Kubernetes Zookeeper Cluster Setup/Configuration YAML

Copy

Download

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zookeeper-sc
  selfLink: /apis/storage.k8s.io/v1/storageclasses/zookeeper-sc
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: 'true'
provisioner: kubernetes.io/azure-disk
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: StandardSSD_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      initContainers:
      - command:
        - /bin/bash
        - -c
        - |-
          set -ex;
          mkdir -p /data;         
          if [[ ! -f "/data/myid" ]]; then
            hostindex=$HOSTNAME; let zooid=${hostindex: -1: 1}+1; echo $zooid > "/data/myid"
            echo "Zookeeper MyId: " $zooid
          fi
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: zookeeper:3.6.2
        name: zookeeper-init
        securityContext:
          runAsUser: 1000
        volumeMounts:
        - name: zk-data
          mountPath: "/data"
      containers:
      - name: zookeeper        
        image: "zookeeper:3.6.2"
        env:           
        - name: ZOO_SERVERS
          value: "server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888;2181 server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888;2181 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888;2181"
        - name: ZOO_STANDALONE_ENABLED
          value: "false"
        - name: ZOO_4LW_COMMANDS_WHITELIST
          value: "srvr,mntr"           
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        volumeMounts:        
        - name: zk-data
          mountPath: "/data"
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: zk-data
    spec:
      storageClassName: "zookeeper-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

Kafka connection refused with Kubernetes nodeport

Copy

Download

apiVersion: v1
kind: Service
metadata:
  name: kafka-internal
  namespace: test
  labels:
    app: kafka-test
    unit: kafka
spec:
  type: NodePort
  selector:
    app: test-app
    unit: kafka
    parentdeployment: test-kafka
  ports:
    - name: kafka
      port: 9092
      protocol: TCP
  type: ClusterIP

apiVersion: v1
kind: Service
metadata:
  name: kafka-external
  namespace: test
  labels:
    app: kafka-test
    unit: kafka
spec:
  type: NodePort
  selector:
    app: test-app
    unit: kafka
    parentdeployment: test-kafka
  ports:
    - name: kafka
      port: 9092
      targetPort: 9092
      protocol: TCP
  type: NodePort

-----------------------

apiVersion: v1
kind: Service
metadata:
  name: kafka-internal
  namespace: test
  labels:
    app: kafka-test
    unit: kafka
spec:
  type: NodePort
  selector:
    app: test-app
    unit: kafka
    parentdeployment: test-kafka
  ports:
    - name: kafka
      port: 9092
      protocol: TCP
  type: ClusterIP

apiVersion: v1
kind: Service
metadata:
  name: kafka-external
  namespace: test
  labels:
    app: kafka-test
    unit: kafka
spec:
  type: NodePort
  selector:
    app: test-app
    unit: kafka
    parentdeployment: test-kafka
  ports:
    - name: kafka
      port: 9092
      targetPort: 9092
      protocol: TCP
  type: NodePort

Spring boot - automatically activate profile based on operating system

Copy

Download

@SpringBootApplication
public class DemoApplication {

  public static void main(String[] args) {
    SpringApplication app = new SpringApplication(DemoApplication.class);
    if (System.getProperty("os.name").contains("darwin")) {
      app.setAdditionalProfiles("mac");
    }
    app.run(args);
  }

}

-----------------------

@Configuration
public class MyWebApplicationInitializer implements WebApplicationInitializer {

@Override
public void onStartup(ServletContext servletContext) throws ServletException {

   if (System.getProperty("os.name").contains("darwin")) {
  servletContext.setInitParameter("spring.profiles.active", "mac");
     }
    
  }
}

SPRING_PROFILES_ACTIVE=mac

-----------------------

@Configuration
public class MyWebApplicationInitializer implements WebApplicationInitializer {

@Override
public void onStartup(ServletContext servletContext) throws ServletException {

   if (System.getProperty("os.name").contains("darwin")) {
  servletContext.setInitParameter("spring.profiles.active", "mac");
     }
    
  }
}

SPRING_PROFILES_ACTIVE=mac

Read data from Kafka and print to console with Spark Structured Sreaming in Python

Copy

Download

spark = SparkSession \
          .builder \
          .appName("APP") \
          .getOrCreate()

df = spark\
      .readStream \
      .format("kafka") \
      .option("kafka.bootstrap.servers", "localhost:9092") \
      .option("subscribe", "sparktest") \
      .option("startingOffsets", "earliest") \
      .load()
      

query = df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
    .writeStream \
    .format("console") \
    .option("checkpointLocation", "path/to/HDFS/dir") \
    .start()

query.awaitTermination()

See all related Code Snippets

Community Discussions

Trending Discussions on zookeeper

Kafka consumer does not print anything
EmbeddedKafka failing since Spring Boot 2.6.X : AccessDeniedException: ..\AppData\Local\Temp\spring.kafka*
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
Define Kafka ACL to limit topic creation
Migrating from FlinkKafkaConsumer to KafkaSource, no windows executed
Kafka integration tests in Gradle runs into GitHub Actions
Accessing HBase on Amazon EMR with Athena
bash + how to capture word from a long output
Cannot connect to kafka from ourside of its docker container
resolve service hostname in ECS fargate task

Trending Discussions on zookeeper

QUESTION

Kafka consumer does not print anything

Asked 2022-Mar-26 at 12:23

I am following this tutorial: https://towardsdatascience.com/kafka-docker-python-408baf0e1088 in order to run a producer-consumer example using Kafka, Docker and Python. My problem is that my terminal prints the iterations of the producer, while it does not print the iterations of consumer. I am running this example in local, so:

in one terminal tab I have done: docker-compose -f docker-compose-expose.yml up where my docker-compose-expose.yml is this:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
     - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
     - "9092:9092"
    expose:
     - "9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "topic_test:1:1"
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock

I see the logs from kafka and zookeeper, so I assume that it is working. 2. In another tab I run python producer.py where my producer.py is this:

from time import sleep
from json import dumps
from kafka import KafkaProducer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda x: dumps(x).encode('utf-8')
)
for j in range(9999):
    print("Iteration", j)
    data = {'counter': j}
    producer.send('topic_test', value=data)
    sleep(0.5)

It prints the iteration, so I think it is working. 3. Finally, in another tab I run python consumer.py where my consumer.py is:

from kafka import KafkaConsumer
from json import loads
from time import sleep
consumer = KafkaConsumer(
    'topic_test',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='my-group-id',
    value_deserializer=lambda x: loads(x.decode('utf-8'))
)
for event in consumer:
    event_data = event.value
    # Do whatever you want
    print(event_data)
    sleep(2)

So, here is the problem. According to the tutorial, it should print something like this:

{'counter': 0}
{'counter': 1}
{'counter': 2}
{'counter': 3}
{'counter': 4}
{'counter': 5}
{'counter': 6}
[...]

However, my terminal does not print anything. Moreover, If I solve this problem I want to run the producer on one machine and the consumer on another machine. I need to change only KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092 in the docker-compose-expose.yml writing the IP address of the producer in place of localhost, right?

UPDATE: I tried on another machine, and these steps are now working. Maybe during my experiments I changed something about kafka or docker, so for this reason it is not working. however I do not know why

ANSWER

Answered 2022-Mar-26 at 12:23

Basically I had understood that the problem was in some images/processes that were in execution. With docker-compose stop and docker-compose rm -f I solved.

Source https://stackoverflow.com/questions/71572487

QUESTION

EmbeddedKafka failing since Spring Boot 2.6.X : AccessDeniedException: ..\AppData\Local\Temp\spring.kafka*

Asked 2022-Mar-25 at 12:39

e: this has been fixed through Spring Boot 2.6.5 (see https://github.com/spring-projects/spring-boot/issues/30243)

Since upgrading to Spring Boot 2.6.X (in my case: 2.6.1), I have multiple projects that now have failing unit-tests on Windows that cannot start EmbeddedKafka, that do run with Linux

There is multiple errors, but this is the first one thrown

...
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.1)

2021-12-09 16:15:00.300  INFO 13864 --- [           main] k.utils.Log4jControllerRegistration$     : Registered kafka:type=kafka.Log4jController MBean
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : 
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :   ______                  _                                          
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :  |___  /                 | |                                         
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :     / /    ___     ___   | | __   ___    ___   _ __     ___   _ __   
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :    / /    / _ \   / _ \  | |/ /  / _ \  / _ \ | '_ \   / _ \ | '__|
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :   / /__  | (_) | | (_) | |   <  |  __/ |  __/ | |_) | |  __/ | |    
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :  /_____|  \___/   \___/  |_|\_\  \___|  \___| | .__/   \___| |_|
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :                                               | |                     
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     :                                               |_|                     
2021-12-09 16:15:00.420  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : 
2021-12-09 16:15:00.422  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2021-12-09 16:15:00.422  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : Server environment:host.name=host.docker.internal
2021-12-09 16:15:00.422  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : Server environment:java.version=11.0.11
2021-12-09 16:15:00.422  INFO 13864 --- [           main] o.a.zookeeper.server.ZooKeeperServer     : Server environment:java.vendor=AdoptOpenJDK
...
2021-12-09 16:15:01.015  INFO 13864 --- [nelReaper-Fetch] lientQuotaManager$ThrottledChannelReaper : [ThrottledChannelReaper-Fetch]: Starting
2021-12-09 16:15:01.015  INFO 13864 --- [lReaper-Produce] lientQuotaManager$ThrottledChannelReaper : [ThrottledChannelReaper-Produce]: Starting
2021-12-09 16:15:01.016  INFO 13864 --- [lReaper-Request] lientQuotaManager$ThrottledChannelReaper : [ThrottledChannelReaper-Request]: Starting
2021-12-09 16:15:01.017  INFO 13864 --- [trollerMutation] lientQuotaManager$ThrottledChannelReaper : [ThrottledChannelReaper-ControllerMutation]: Starting
2021-12-09 16:15:01.037  INFO 13864 --- [           main] kafka.log.LogManager                     : Loading logs from log dirs ArraySeq(C:\Users\ddrop\AppData\Local\Temp\spring.kafka.bf8e2b62-a1f2-4092-b292-a15e35bd31ad18378079390566696446)
2021-12-09 16:15:01.040  INFO 13864 --- [           main] kafka.log.LogManager                     : Attempting recovery for all logs in C:\Users\ddrop\AppData\Local\Temp\spring.kafka.bf8e2b62-a1f2-4092-b292-a15e35bd31ad18378079390566696446 since no clean shutdown file was found
2021-12-09 16:15:01.043  INFO 13864 --- [           main] kafka.log.LogManager                     : Loaded 0 logs in 6ms.
2021-12-09 16:15:01.043  INFO 13864 --- [           main] kafka.log.LogManager                     : Starting log cleanup with a period of 300000 ms.
2021-12-09 16:15:01.045  INFO 13864 --- [           main] kafka.log.LogManager                     : Starting log flusher with a default period of 9223372036854775807 ms.
2021-12-09 16:15:01.052  INFO 13864 --- [           main] kafka.log.LogCleaner                     : Starting the log cleaner
2021-12-09 16:15:01.059  INFO 13864 --- [leaner-thread-0] kafka.log.LogCleaner                     : [kafka-log-cleaner-thread-0]: Starting
2021-12-09 16:15:01.224  INFO 13864 --- [name=forwarding] k.s.BrokerToControllerRequestThread      : [BrokerToControllerChannelManager broker=0 name=forwarding]: Starting
2021-12-09 16:15:01.325  INFO 13864 --- [           main] kafka.network.ConnectionQuotas           : Updated connection-accept-rate max connection creation rate to 2147483647
2021-12-09 16:15:01.327  INFO 13864 --- [           main] kafka.network.Acceptor                   : Awaiting socket connections on localhost:63919.
2021-12-09 16:15:01.345  INFO 13864 --- [           main] kafka.network.SocketServer               : [SocketServer listenerType=ZK_BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT)
2021-12-09 16:15:01.350  INFO 13864 --- [0 name=alterIsr] k.s.BrokerToControllerRequestThread      : [BrokerToControllerChannelManager broker=0 name=alterIsr]: Starting
2021-12-09 16:15:01.364  INFO 13864 --- [eaper-0-Produce] perationPurgatory$ExpiredOperationReaper : [ExpirationReaper-0-Produce]: Starting
2021-12-09 16:15:01.364  INFO 13864 --- [nReaper-0-Fetch] perationPurgatory$ExpiredOperationReaper : [ExpirationReaper-0-Fetch]: Starting
2021-12-09 16:15:01.365  INFO 13864 --- [0-DeleteRecords] perationPurgatory$ExpiredOperationReaper : [ExpirationReaper-0-DeleteRecords]: Starting
2021-12-09 16:15:01.365  INFO 13864 --- [r-0-ElectLeader] perationPurgatory$ExpiredOperationReaper : [ExpirationReaper-0-ElectLeader]: Starting
2021-12-09 16:15:01.374  INFO 13864 --- [rFailureHandler] k.s.ReplicaManager$LogDirFailureHandler  : [LogDirFailureHandler]: Starting
2021-12-09 16:15:01.390  INFO 13864 --- [           main] kafka.zk.KafkaZkClient                   : Creating /brokers/ids/0 (is it secure? false)
2021-12-09 16:15:01.400  INFO 13864 --- [           main] kafka.zk.KafkaZkClient                   : Stat of the created znode at /brokers/ids/0 is: 25,25,1639062901396,1639062901396,1,0,0,72059919267528704,204,0,25

2021-12-09 16:15:01.400  INFO 13864 --- [           main] kafka.zk.KafkaZkClient                   : Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://localhost:63919, czxid (broker epoch): 25
2021-12-09 16:15:01.410 ERROR 13864 --- [           main] kafka.server.BrokerMetadataCheckpoint    : Failed to write meta.properties due to

java.nio.file.AccessDeniedException: C:\Users\ddrop\AppData\Local\Temp\spring.kafka.bf8e2b62-a1f2-4092-b292-a15e35bd31ad18378079390566696446
    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89) ~[na:na]
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[na:na]
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) ~[na:na]

Reproduceable via spring Initializr + adding "Spring Kafka": https://start.spring.io/#!type=maven-project&language=java&platformVersion=2.6.1&packaging=jar&jvmVersion=11&groupId=com.example&artifactId=demo&name=demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.example.demo&dependencies=kafka

And then have following test-class to execute:

package com.example.demo;

import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.test.context.EmbeddedKafka;

@SpringBootTest
@EmbeddedKafka
class ApplicationTest {

    @Test
    void run() {
        int i = 1 + 1; // just a line of code to set a debug-point
    }
}

I do not have this error when pinning kafka.version to 2.8.1 in pom.xml's properties.

It seems like the cause is in Kafka itself, but I have a hard time figuring out if it is spring-kafka intitializing Kafka via EmbeddedKafka incorrectly or if Kafka itself is the culrit here.

Anyone has an idea? Am I missing a test-parameter to set?

ANSWER

Answered 2021-Dec-09 at 15:51

Known bug on the Apache Kafka side. Nothing to do from Spring perspective. See more info here: https://github.com/spring-projects/spring-kafka/discussions/2027. And here: https://issues.apache.org/jira/browse/KAFKA-13391

You need to wait until Apache Kafka 3.0.1 or don't use embedded Kafka and just rely on the Testcontainers, for example, or fully external Apache Kafka broker.

Source https://stackoverflow.com/questions/70292425

QUESTION

Define Kafka ACL to limit topic creation

Asked 2021-Dec-30 at 07:35

We are currently running an unsecured Kafka setup on AWS MSK (so I don't have access to most config files directly and need to use the kafka-cli) and are looking into ways to add protection. Setting up TLS & SASL is easy, though as our Kafka cluster is behind a VPN and already has restricted access does not add more security.

We want to start with the most important and in our opinion quick win security addition. Protect topics from being deleted (and created) by all users. We currently have allow.everyone.if.no.acl.found set to true.

All I find on Google or Stack Overflow shows me how I can restrict users from reading/writing to other topics than they have access to. Though Ideally that is not what we want to implement as a first step.

I have found things about a root-user (Is an admin user, though was called root in all tutorials I read). Though the examples I have found don't show examples of adding an ACL to this root user to make it the only one accessible, the topic deletion/creation.

Can you please explain how to create a user that, and block all other users?

By the way, we also don't use zookeeper, even though an MSK-cluster ads this per default. And hope we can do this without adding zookeeper actively to our stack. The answer given here hardly relies on zookeeper. Also, this answer points to the topic read/write examples only, even though the question was the same as I am asking

ANSWER

Answered 2021-Dec-21 at 10:11

I'd like to start with a disclaimer that I'm personally not familiar with AWS MSK offering in great detail so this answer is largely based on my understanding of the open source distribution of Apache Kafka.

First - The Kafka ACLs are actually stored in Zookeeper by default so if you're not using Zookeeper, it might be worth adding this if you're not using it.

Reference - Kafka Definitive Guide - 2nd edition - Chapter 11 - Securing Kafka - Page 294

Second - If you're using SASL for authentication through any of the supported mechanisms such as GSSAPI (Kerberos), then you'll need to create a principal as you would normally create one and use one of the following options:

Add the required permissions for topic creation/deletion etc. using the kafka-acls command (Command Reference)

bin/kafka-acls.sh --add --cluster --operation Create --authorizer-properties zookeeper.connect=localhost:2181 --allow-principal User:admin

Note - admin is the assumed principal name
Or add admin user to the super users list in server.properties file by adding the following line so it has unrestricted access on all resources

super.users=User:Admin

Any more users can be added in the same line delimited by ;.

To add the strictness, you'll need to set allow.everyone.if.no.acl.found to false so any access to any resources is only granted by explicitly adding these permissions.

Third - As you've asked specifically about your root user, I'm assuming you're referring to the linux root here. You could just restrict the linux level permissions using chmod command for the kafka-acls.sh script but that is quite a crude way of achieving what you need. I'm also not entirely sure if this is doable in MSK or not.

Source https://stackoverflow.com/questions/70409488

QUESTION

Migrating from FlinkKafkaConsumer to KafkaSource, no windows executed

Asked 2021-Nov-30 at 05:29

I am a kafka and flink beginner. I have implemented FlinkKafkaConsumer to consume messages from a kafka-topic. The only custom setting other than "group" and "topic" is (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") to enable re-reading the same messages several times. It works out of the box for consuming and logic. Now FlinkKafkaConsumer is deprecated, and i wanted to change to the successor KafkaSource.

Initializing KafkaSource with the same parameters as i do FlinkKafkaConsumer produces a read of the topic as expected, i can verify this by printing the stream. De-serialization and timestamps seem to work fine. However execution of windows are not done, and as such no results are produced.

I assume some default setting(s) in KafkaSource are different from that of FlinkKafkaConsumer, but i have no idea what they might be.

KafkaSource - Not working

 KafkaSource<TestData> source = KafkaSource.<TestData>builder()
                .setBootstrapServers(propertiesForKafka.getProperty("bootstrap.servers"))
                .setTopics(TOPIC)
                .setDeserializer(new CustomDeserializer())
                .setGroupId(GROUP_ID)
                .setStartingOffsets(OffsetsInitializer.earliest())
                .build();

        DataStream<TestData> testDataStreamSource = env.fromSource(
                source,
                WatermarkStrategy.
                        <TestData>noWatermarks(),
                "Kafka Source"
        );

Kafka consumer - Working (Properties contains group.id,bootstrap.servers and zookeeper.connect)

    propertiesForKafka.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    FlinkKafkaConsumer<TestData> flinkKafkaConsumer = new  FlinkKafkaConsumer(TOPIC, new CustomDeserializer(), propertiesForKafka);
    DataStreamSource<TestData> testDataStreamSource = env.addSource(flinkKafkaConsumer)

Both streams use the same pipeline that looks like this

testDataStreamSource
           .assignTimestampsAndWatermarks(WatermarkStrategy.
                        <TestData>forMonotonousTimestamps().
                        withTimestampAssigner((event, timestamp) -> event.getTimestamp()))
            .keyBy(TestData::getKey)
            .window(SlidingEventTimeWindows.of(Time.hours(3), Time.hours(1)))
            .process(new ProcessWindowFunction<TestData, TestDataOutput, String, TimeWindow>() {
                    @Override
                    public void process(
            ....
            });

Things tried

I've tried to experiment with setting committing of offsets, but it has not improved the situation.
Setting timestamps already in the source.

ANSWER

Answered 2021-Nov-24 at 18:39

Update: The answer is that the KafkaSource behaves differently than FlinkKafkaConsumer in the case where the number of Kafka partitions is smaller than the parallelism of Flink's kafka source operator. See https://stackoverflow.com/a/70101290/2000823 for details.

Original answer:

The problem is almost certainly something related to the timestamps and watermarks.

To verify that timestamps and watermarks are the problem, you could do a quick experiment where you replace the 3-hour-long event time sliding windows with short processing time tumbling windows.

In general it is preferred (but not required) to have the KafkaSource do the watermarking. Using forMonotonousTimestamps in a watermark generator applied after the source, as you are doing now, is a risky move. This will only work correctly if the timestamps in all of the partitions being consumed by each parallel instance of the source are processed in order. If more than one Kafka partition is assigned to any of the KafkaSource tasks, this isn't going to happen. On the other hand, if you supply the forMonotonousTimestamps watermarking strategy in the fromSource call (rather than noWatermarks), then all that will be required is that the timestamps be in order on a per-partition basis, which I imagine is the case.

As troubling as that is, it's probably not enough to explain why the windows don't produce any results. Another possible root cause is that the test data set doesn't include any events with timestamps after the first window, so that window never closes.

Do you have a sink? If not, that would explain things.

You can use the Flink dashboard to help debug this. Look to see if the watermarks are advancing in the window tasks. Turn on checkpointing, and then look to see how much state the window task has -- it should have some non-zero amount of state.

Source https://stackoverflow.com/questions/69765972

QUESTION

Kafka integration tests in Gradle runs into GitHub Actions

Asked 2021-Nov-03 at 19:11

We've been moving our applications from CircleCI to GitHub Actions in our company and we got stuck with a strange situation.

There has been no change to the project's code, but our kafka integration tests started to fail in GH Actions machines. Everything works fine in CircleCI and locally (MacOS and Fedora linux machines).

Both CircleCI and GH Actions machines are running Ubuntu (tested versions were 18.04 and 20.04). MacOS was not tested in GH Actions as it doesn't have Docker in it.

Here are the docker-compose and workflow files used by the build and integration tests:

docker-compose.yml

version: '2.1'

services:
  postgres:
    container_name: listings-postgres
    image: postgres:10-alpine
    mem_limit: 500m
    networks:
      - listings-stack
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: listings
      POSTGRES_PASSWORD: listings
      POSTGRES_USER: listings
      PGUSER: listings
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 1s
      timeout: 3s
      retries: 30

  listings-zookeeper:
    container_name: listings-zookeeper
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - listings-stack
    ports:
      - "2181:2181"
    healthcheck:
      test: nc -z localhost 2181 || exit -1
      interval: 10s
      timeout: 5s
      retries: 10

  listings-kafka:
    container_name: listings-kafka
    image: confluentinc/cp-kafka:6.2.0
    depends_on:
      listings-zookeeper:
        condition: service_healthy
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://listings-kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: listings-zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - listings-stack
    ports:
      - "29092:29092"
    healthcheck:
      test: kafka-topics --bootstrap-server 127.0.0.1:9092 --list
      interval: 10s
      timeout: 10s
      retries: 50

networks: {listings-stack: {}}

build.yml

name: Build

on: [ pull_request ]

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.TUNNEL_AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.TUNNEL_AWS_SECRET_ACCESS_KEY }}
  AWS_DEFAULT_REGION: 'us-east-1'
  CIRCLECI_KEY_TUNNEL: ${{ secrets.ID_RSA_CIRCLECI_TUNNEL }}

jobs:
  build:
    name: Listings-API Build
    runs-on: [ self-hosted, zap ]

    steps:
      - uses: actions/checkout@v2
        with:
          token: ${{ secrets.GH_OLXBR_PAT }}
          submodules: recursive
          path: ./repo
          fetch-depth: 0

      - name: Set up JDK 11
        uses: actions/setup-java@v2
        with:
          distribution: 'adopt'
          java-version: '11'
          architecture: x64
          cache: 'gradle'

      - name: Docker up
        working-directory: ./repo
        run: docker-compose up -d

      - name: Build with Gradle
        working-directory: ./repo
        run: ./gradlew build -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -x integrationTest

      - name: Integration tests with Gradle
        working-directory: ./repo
        run: ./gradlew integrationTest -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

      - name: Sonarqube
        working-directory: ./repo
        env:
          GITHUB_TOKEN: ${{ secrets.GH_OLXBR_PAT }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
        run: ./gradlew sonarqube --info -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

      - name: Docker down
        if: always()
        working-directory: ./repo
        run: docker-compose down --remove-orphans

      - name: Cleanup Gradle Cache
        # Remove some files from the Gradle cache, so they aren't cached by GitHub Actions.
        # Restoring these files from a GitHub Actions cache might cause problems for future builds.
        run: |
          rm -f ${{ env.HOME }}/.gradle/caches/modules-2/modules-2.lock
          rm -f ${{ env.HOME }}/.gradle/caches/modules-2/gc.properties

The integration tests are written using the Spock framework and the part where the errors occur are these:

  boolean compareRecordSend(String topicName, int expected) {
    def condition = new PollingConditions()
    condition.within(kafkaProperties.listener.pollTimeout.getSeconds() * 5) {
      assert expected == getRecordSendTotal(topicName)
    }
    return true
  }

  int getRecordSendTotal(String topicName) {
    kafkaTemplate.flush()
    return kafkaTemplate.metrics().find {
      it.key.name() == "record-send-total" && it.key.tags().get("topic") == topicName
    }?.value?.metricValue() ?: 0
  }

The error we're getting is:

Condition not satisfied after 50.00 seconds and 496 attempts
    at spock.util.concurrent.PollingConditions.within(PollingConditions.java:185)
    at com.company.listings.KafkaAwareBaseSpec.compareRecordSend(KafkaAwareBaseSpec.groovy:31)
    at com.company.listings.application.worker.listener.notifier.ListingNotifierITSpec.should notify listings(ListingNotifierITSpec.groovy:44)

    Caused by:
    Condition not satisfied:

    expected == getRecordSendTotal(topicName)
    |        |  |                  |
    10       |  0                  v4
                false

We've debugged the GH Actions machine (SSH into it) and run things manually. The error still happens, but if the integration tests are run a second time (as well as subsequent runs), everything works perfectly.

We've also tried to initialize all the necessary topics and send some messages to them preemptively, but the behavior was the same.

The questions we have are:

Is there any issue when running Kafka dockerized in an Ubuntu machine (the error also occurred in a co-worker Ubuntu machine)?
Any ideas on why this is happening?

Edit

application.yml (Kafka related configuration)

spring:
  kafka:
    bootstrap-servers: localhost:29092
    producer:
      batch-size: 262144
      buffer-memory: 536870912
      retries: 1
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.ByteArraySerializer
      acks: all
      properties:
        linger.ms: 0

ANSWER

Answered 2021-Nov-03 at 19:11

We identified some test sequence dependency between the Kafka tests.

We updated our Gradle version to 7.3-rc-3 which has a more deterministic approach to test scanning. This update "solved" our problem while we prepare to fix the tests' dependencies.

Source https://stackoverflow.com/questions/69284830

QUESTION

bash + how to capture word from a long output

Asked 2021-Sep-03 at 07:02

I have the following output from the following command

zookeeper-shell.sh 19.2.6.4  get /brokers/ids/1010

output is

Connecting to 19.2.6.4

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://kafka1.lulu.com:6667"],"rack":"/default-rack","jmx_port":9997,"port":6667,"host":"kafka1.lulu.com","version":4,"timestamp":"1630507307906"}

The main target is to capture the machine name kafka1 – from above output

So I successes to do the following long command syntax

zookeeper-shell.sh 119.2.6.4  get /brokers/ids/1010 | sed s'/\/\// /g' | sed s'/:/ /g' | sed s'/,/ /g' | sed s'/"/ /g' | sed s'/\./ /g'| awk '{for (i=1;i<=NF;i++) print $i}' | grep -i kafka | sort | uniq

and the results is: ( as expected results )

kafka1

the case is that I feel bad with my approach , its too long and not so elegant

can we get suggestion ( with awk/sed/perl one liner ) , that are much better from my syntax?

ANSWER

Answered 2021-Sep-02 at 17:28

With your shown samples, attempts please try following awk code. Since I don't have zookeeper command with me, I had written this code and tested it as per your shown output only.

zookeeper-shell.sh 19.2.6.4  get /brokers/ids/1010 | 
awk '
/WatchedEvent state/{
  found=1
  next
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){
  print substr($0,RSTART+13,RLENGTH-13)
}
'

Explanation: Adding detailed explanation for above awk code.

awk '                                         ##Starting awk program from here.
/WatchedEvent state/{                         ##Checking condition if line contains WatchedEvent state
  found=1                                     ##Then set found to 1 here.
  next                                        ##next will skip all further statements from here.
}
found && match($0,/"PLAINTEXT:\/\/[^:]*/){    ##Checking condition if found is SET then match regex "PLAINTEXT:\/\/[^:]* in match function of awk.
  print substr($0,RSTART+13,RLENGTH-13)       ##Printing sub string of matched regex used in match function above.
}
'

Source https://stackoverflow.com/questions/69034663

QUESTION

Cannot connect to kafka from ourside of its docker container

Asked 2021-Aug-30 at 00:40

This is my docker compose file:

version: "3.7"

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports: 
      - 2181:2181
    env_file: 
      - zookeeper.env
  
  kafka:
    image: 'bitnami/kafka:latest'
    container_name: kafka
    env_file:
      - kafka.env
    ports: 
      - 9093:9092
    depends_on:
      - zookeeper

And also these are my .env files:

kakfa.env

KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
KAFKA_CFG_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://localhost:9093
KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://localhost:9093
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=1
ALLOW_PLAINTEXT_LISTENER=yes

zookeeper.env

ZOOKEEPER_CLIENT_PORT=2181
ZOOKEEPER_TICK_TIME=2000
ALLOW_ANONYMOUS_LOGIN=yes

When I make the compose file up, everything seems to be fine and I have good telnet result form outside of the docker container, on my localhost, to 9093 port.

I write a simple producer python file as below:

import logging

from kafka import KafkaProducer

logging.basicConfig(level=logging.INFO)


class Producer():
    def __init__(self) -> None:
        self.conn = KafkaProducer(bootstrap_servers="localhost:9093")

    def produce(self):
        while True:
            try:
                self.conn.send("test", b'test')
            except KeyboardInterrupt:
                logging.info("Producer finished")
                break


if __name__ == "__main__":
    producer = Producer()
    producer.produce()

When I run my code, I get this error:

WARNING:kafka.conn:DNS lookup failed for kafka:9092, exception was [Errno -3] Temporary failure in name resolution. Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and resolvable?
ERROR:kafka.conn:DNS lookup failed for kafka:9092 (AddressFamily.AF_UNSPEC)

I read also this post but I could not resolve my error and I don't know what should I do to get rid of this error.

ANSWER

Answered 2021-Aug-30 at 00:39

You've forwarded the wrong port

9093 on the host needs to map to the localhost:9093 advertised port

Otherwise, you're connecting to 9093, which returns kafka:9092, as explained in the blog. Container hostnames cannot be resolved by the host, by default

Source https://stackoverflow.com/questions/68975387

QUESTION

resolve service hostname in ECS fargate task

Asked 2021-Aug-22 at 20:28

I am trying to automate my ECS fargate cluster making using terraform.

I have a SpringBoot project with microservices containerized, and I am putting these images in a single task definition for an ECS service for the backend.

The ECS cluster is initially running, but Kafka is getting stopped with the error :

ERROR Unable to resolve address: zookeeper:2181
(org.apache.zookeeper.client.StaticHostProvider)

I have given the image for zookeeper(bitnami) and used the KAFKA_CFG_ZOOKEEPER_CONNECT environment variable too.

EDIT : My task definition:

resource "aws_ecs_task_definition" "this" {
  family                   = local.application_name
  requires_compatibilities = [local.launch_type]
  execution_role_arn       = data.aws_iam_role.ecs_task_execution_role.arn
  network_mode             = "awsvpc"
  cpu                      = "4096"
  memory                   = "30720"
  container_definitions = jsonencode([
    {
      name      = "zookeeper"
      image     = "docker.io/bitnami/zookeeper:latest"
      essential = true
      cpu       = 512
      memory    = 1024
      portMappings = [
        {
          containerPort = 2181
          hostPort      = 2181
        }
      ]
      "environment" : [
        { "name" : "ALLOW_ANONYMOUS_LOGIN", "value" : "yes" },
        { "name" : "ZOO_LISTEN_ALLIPS_ENABLED", "value" : "yes" }
      ]
    },
    {
      name      = "kafka-server"
      image     = "docker.io/bitnami/kafka:latest"
      essential = true
      cpu       = 512
      memory    = 1024
      portMappings = [
        {
          containerPort = 9092
          hostPort      = 9092
        }
      ]
      depends_on =  [
        "zookeeper"
      ]
      "environment" : [
        { "name" : "KAFKA_CFG_ZOOKEEPER_CONNECT", "value" : "zookeeper:2181" },
        { "name" : "ALLOW_ANONYMOUS_LOGIN", "value" : "yes" },
        { "name" : "ALLOW_PLAINTEXT_LISTENER", "value" : "yes" }
      ],
    },
    {
      name      = "email-service"
      image     = "my-email-image"
      essential = true
      cpu       = 512
      memory    = 1024
      portMappings = [
        {
          containerPort = 8090
          hostPort      = 8090
        }
      ]
      "environment" : [
        { "name" : "EMAIL_URL", "value" : "email-service" },
        { "name" : "EMAIL_PORT", "value" : "8090" },
        { "name" : "KAFKA_URL", "value" : "kafka-server" },
        { "name" : "KAFKA_PORT", "value" : "9092" },
      ]
    }
  ])
}

ANSWER

Answered 2021-Aug-22 at 17:28

AS written in the documentation:

Additionally, containers that belong to the same task can communicate over the localhost interface.

So my suggestion is to use localhost instead of the service names. For example, you want to do it for Kafka but also for every other service, such as the email service.

{ "name" : "KAFKA_CFG_ZOOKEEPER_CONNECT", "value" : "localhost:2181" },

Source https://stackoverflow.com/questions/68880004

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install zookeeper

You can download it from GitHub, Maven.
You can use zookeeper like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the zookeeper component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

We always welcome new contributors to the project! See How to Contribute for details on how to submit patches as pull requests and other aspects of our contribution workflow.