kcl | Your one stop shop to do anything with Kafka. Producing, consuming, transacting, administrating; 0.8 | Pub Sub library
kandi X-RAY | kcl Summary
kandi X-RAY | kcl Summary
kcl is a complete, pure Go command line Kafka client. Think of it as your one stop shop to do anything you want to do with Kafka. Producing, consuming, transacting, administrating, and so on. Unlike the small size of [kafkacat][1], this binary is ~12M compiled. It is, however, still fast, has rich consuming and producing formatting options, and a complete Kafka administration interface.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- unstick unstick a topic to a given topic
- ParseWriteFormat takes a format string and parses it into a format function .
- listOffsetsCommand returns a cobra . Command for list offsets
- alterUserSCRAM is a command to alter a user scram .
- alterClientQuotas returns a cobra command for update client quotas
- describeClientQuotas returns a cobra command for describe client quotas
- deleteRecordsCommand returns a cobra . Command for delete records
- parseReadSize takes a format string and returns a function that can be used to parse the size of the given format .
- createCommand returns the cobra command for creating a new ACL
- transact runs in a separate goroutine
kcl Key Features
kcl Examples and Code Snippets
Lines of Code : 5License : Permissive (BSD-3-Clause)
Copyecho fubar | kcl produce foo
kcl produce foo < baz
echo barfoo | kcl produce foo -f'%K{3}%V{3}%v%k\n'
echo "key: bizzy, value: bazzy" | kcl produce foo -f 'key: %k, value: %v\n'
echo "1 k 1 v 2 2 h1 2 v1 2 h2 2 v2 " | kcl produce foo -f '%K %k %V kcl consume foo
kcl consume foo -f "KEY=%k, VALUE=%v, HEADERS=%{%h{ '%k'='%v' }}\n"
kcl consume -g grup foo bar Community Discussions
Trending Discussions on kcl
QUESTION
I am evaluating different streaming/messaging services for use as an Event Bus. One of the dimensions I am considering is the ordering guarantee provided by each of the services. Two of the options that I am exploring are AWS Kinesis and Kafka and from a high level, according it looks like they both provide similar ordering guarantees where records are guaranteed to be consumable in the same order they were published only within that shard/partition.
It seems that AWS Kinesis APIs expose the ids of the parent shard(s) such that Consumer Groups using KCL can ensure records with the same partition key can be consumed in the order they were published (assuming a single threaded publisher) even if shards are being split and merged.
My question is, does Kafka provide any similar functionality such that records published with a specific key can be consumed in order even if partitions are added while messages are being published? From my reading, my understanding of partition selection (if you are specifying keys with your records) behaves along the lines of HASH(key) % PARTITION_COUNT. So, if additional partitions are added, they partition where all messages with a specific key will be published may (and I've proven it does locally) change. Simultaneously, the Group Coordinator/Leader will reassign partition ownership among Consumers in Consumer Groups receiving records from that topic. But, after reassignment, there will be records (potentially unconsumed records) with the same key found in two different partitions. So, from the Consumer Group level is there no way to ensure that the unconsumed records with the same key now found in different partitions will be consumed in the order they were published?
I have very little experience with both these services, so my understanding may be flawed. Any advice is appreciated!
...ANSWER
Answered 2022-Jan-31 at 22:21My understanding was correct (as confirmed by @OneCricketeer and the documentation). Here is the relevant section of the documentation:
Although it’s possible to increase the number of partitions over time, one has to be careful if messages are produced with keys. When publishing a keyed message, Kafka deterministically maps the message to a partition based on the hash of the key. This provides a guarantee that messages with the same key are always routed to the same partition. This guarantee can be important for certain applications since messages within a partition are always delivered in order to the consumer. If the number of partitions changes, such a guarantee may no longer hold. To avoid this situation, a common practice is to over-partition a bit. Basically, you determine the number of partitions based on a future target throughput, say for one or two years later. Initially, you can just have a small Kafka cluster based on your current throughput. Over time, you can add more brokers to the cluster and proportionally move a subset of the existing partitions to the new brokers (which can be done online). This way, you can keep up with the throughput growth without breaking the semantics in the application when keys are used.
QUESTION
I am using the Spring Cloud Stream Binder for Kinesis with KPL/KCL enabled. We would like to disable Cloudwatch metrics without having to manage the configuration of KPL and KCL ourselves (completely overriding the beans). We would like to use the same bean definition for the KinesisProducerConfiguration and each of the KinesisClientLibConfiguration besides the KinesisProducerConfiguration.setMetricsLevel() and KinesisClientLibConfiguration.withMetricsLevel(...) properties.
For reference, here is where the AWS beans are defined in the Spring Cloud Stream Kinesis Binder: KinesisBinderConfiguration.java
What would be the most effective way to do this?
Any help is appreciated! Thanks.
...ANSWER
Answered 2021-Dec-07 at 21:33The framework does not provide any of the KinesisClientLibConfiguration. It is your project responsibility to expose such a bean and with whatever options you need: https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/blob/main/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc#kinesis-consumer-properties
Starting with version 2.0.1, beans of
KinesisClientLibConfigurationtype can be provided in the application context to have a full control over Kinesis Client Library configuration options.
The producer side indeed is covered by the KinesisProducerConfiguration bean in the KinesisBinderConfiguration:
QUESTION
I tried to build up a Jekyll blog website using GitHub pages. I could check the homepage, but the subpages (about & blogposts) showed 404.
To find out where the problem is, I opened a new repo. And I set up the basic things of a Jekyll site using jekyll new . locally and uploaded them to the Github repo. I did not change anything after this step.
And then, I used jekyll serve to run the local test, and everything went well. The layout looked nice and I could check the first blog "Welcome to Jekyll!"(built by default).
However, when I used the link of GitHub Pages to check, the layout of the homepage looked quite different, and I could not check the default blogpost "Welcome to Jekyll!", which showed me 404.
How can I fix it?
This is my repo: https://github.com/jl-xie-kcl/blog20211122
(you can check the screenshots in issue 2 https://github.com/jl-xie-kcl/blog20211122/issues/2)
...ANSWER
Answered 2021-Nov-22 at 11:38Those pages do work, your links are just incorrect because your blog is not at the root of your domain — and this goes the same for the style and images not working, by the way:
- The link to your about page: https://jl-xie-kcl.github.io/blog20211122/about
- The link to your blog post: https://jl-xie-kcl.github.io/blog20211122/jekyll/update/2021/11/22/welcome-to-jekyll.html
In order to fix this, you will have to change the baseurl value in your _config.yml to:
QUESTION
I am facing the following issue when running a KCL consumer against a LocalStack instance:
...ANSWER
Answered 2021-Sep-01 at 19:02Turns out this was much simpler than I thought. I failed to correctly set up the Scheduler.
QUESTION
I don't understand why I can't calculate the average value from the list.
When I put the values for this function in print, it works.
When I use print for sum and len from list a, it works.
But when I try to substitute values from a list into my function, it doesn't work.
I don't understand why.
...ANSWER
Answered 2021-Aug-10 at 19:35I'd recommend using statistics.mean instead of implementing your own average function:
QUESTION
I am using Spring Cloud Stream Kinesis binder (version 2.1.0)
Because of security reasons, I must have one set of credentials for Kinesis and another set of credentials for DynamoDB and CloudWatch.
Everything works fine if spring.cloud.stream.kinesis.binder.kplKclEnabled is set to false. But if it is set to true I have the exception
ANSWER
Answered 2021-Mar-11 at 15:48Your configuration is correct: if you need to use different credentials for those services, you definitely need to declare custom beans for them. The DynamoDB and CloudWatch are required services for Kinesis Client Library. It is used from one hand to manage an offset from stream shards, and on the other - to handle consumer instances changes in the cluster for shards exclusive access. So, it's indeed the fact that Kinesis resource must be available for DynamoDB and CloudWatch users.
See more info in Kinesis Client Library or ask AWS support: nothing Kinesis Binder can do for you on the matter...
https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html
QUESTION
Im trying to work out the correct syntax based on the below if statement in a jenkinsfile, however its not passing the wildcard as expected and trying to match the syntax including the wildcard symbol:
...ANSWER
Answered 2021-Feb-17 at 08:04I would recommend using groovy code instead of a shell script.
QUESTION
There's this document on AWS that suggests the best way to consume a dynamoDB Stream is via Kinesis Client Library using an adapter to translate between kinesis stream and dynamodb stream API.
This is the document: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html
And these are maven coordinates for the adapter implementation
...ANSWER
Answered 2020-Dec-03 at 09:09Answering my own question after researching a bit more.
There seem to be no equivalent of Dynamodb Streams adapter for KCL 2.x and Java SDK 2.x, so you'll need to roll out your own if you want to consume dynamodb stream with KCL 2.x.
Also, around 2 weeks ago a new feature has been added to Dynamodb that allows to stream item changes directly to Kinesis streams. This then allows to use KCL 2.x without any adapters. https://aws.amazon.com/about-aws/whats-new/2020/11/now-you-can-use-amazon-kinesis-data-streams-to-capture-item-level-changes-in-your-amazon-dynamodb-table/
QUESTION
The problem is not related to the number of the CPU assigned to the job. Before this problem, I had an error with the Nvidia driver configuration in a way that I couldn't detect the GPUs by 'nvidia-smi', after solving that error by running 'NVIDIA-Linux-x86_64-410.79.run --no-drm' I have encountered this error. Appreciate very much any help!
PS Before the first problem, I could run similar jobs smoothly
...ANSWER
Answered 2020-Sep-25 at 08:05You sinfo command reports the node as down*, which means it is marked as down by slurm and the slurmd is not reachable. So there is definitely something wrong with the node, which you cannot solve from the user side.
QUESTION
We are building a service based on Kinesis / DynamoDB streams and we have the following question about the behavior of the checkpoints.
We have a worker that starts with the following configuration withInitialPositionInStream (InitialPositionInStream.LATEST) and the name of the KCL application is always the same.
What we have observed by turning the worker off and on again is that it does not start to consume from the end of the stream, since we have a lag metric and we see that when the worker is turned on the consumption lag is hours, when we expect it to be less of 1 second since they are messages that we produce at the moment.
- Is this an expected behavior?
- Are we misinterpreting how the
LATESTworks?
Thank you very much.
...ANSWER
Answered 2020-Oct-06 at 14:14As the documentation for InitialPositionInStream states,
Used to specify the position in the stream where a new application should start from. This is used during initial application bootstrap (when a checkpoint doesn't exist for a shard or its parents).
So, it's used only during initial new application bootstrap and in case of LATEST, it starts after the most recent data record. But only when a checkpoint doesn't exist for a shard or its parents.
So, if you turn your worker off and then turn it on again, it's not expected to start from LATEST anymore but instead it starts from the last checkpointed sequence number for a shard.
KCL does not checkpoint automatically and thus if your worker starts with an hours lag means that probably you checkpoint too rare.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install kcl
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
