flume | A safe and fast multi-producer , multi-consumer channel

Your deployment on Apache Flume is using a version of Jetty older than 9.4.29 (where Attributes.unwrap() was first introduced).

Double check the startup logs of the Jetty server, it will announce the version it thinks it is running.

Example:

1. Can I consume message foo again with another consumer process somewhere else?

Yes. It can be consumed as many times as we want, either by using a new consumer group or by resetting the offset of the existing consumer group.

2.To the opposite,I don't want to consume message foo again, so when I try to consume with another consumer,I should obtain nothing.How can I achieve this?

It's all tied to a consumer group name, which typically tied to one application that is needing these messages. we need to keep the same consumer group name, and typically commited offset is retained for a week(can be changed), so, we can run the app n no of times from n different places by keeping same consumer group name, we will not consume it again, unless we reset the offset.

3.What if there are two messages foo and bar.Can I specify consumers precisely?(For example,I want process A consumes message foo and process B consumes message bar.To go further ,can O specify a range of message offsets? Does it have something to do with consumer group?

We can always consume particular message of a given partition and offset by positioning the consumer group at that offset. Its called seeking an offset, rather than seeking to earliest or latest.

I'm sure you won't be able to do that. Because ambari will need to download the repo from hortonwork. Eventually it will still fail, since there is no public for the hortonwork from 31/01/2021.

hadoop fs -mkdir "/wimp/contract-snapshot/year=2020"

Adding quotation solves the problem.

I managed to make it works. For those who want to know, please read this.

Firstly, change the Flume version. I use now flume 1.7.0 https://flume.apache.org/releases/1.7.0.html. But maybe a newer version would work, I don't want to break it down :)

Secondly, clone this repo https://github.com/cloudera/cdh-twitter-example. Inside, there is a flume.conf file. I configured it like that :

As the error says, r1 has no type, so it doesn't know what to do with your source

You're missing

The error says

It is a single argLine, like:

In case someone stumbles upon this issue... After some further looking into I believe the Apache Flume project is dead in the water and we're going to stop using it. For details please see the following:

• https://medium.com/@bmedalen/great-post-again-prathamesh-i-really-enjoy-your-content-posts-cc0b831a3811

• Many open issues… no responses… https://cwiki.apache.org/confluence/display/FLUME/Developer+Section

• Last version released almost 2 years ago (in the beginning it used to release every 6 months). https://flume.apache.org/index.html

• Wiki not even updated with last versions - last update more then 5 years ago - https://cwiki.apache.org/confluence/display/FLUME/Home

Sadly I'm going to have to close this question in this way. Hope that helps anyone.