fetchers | Subelsky Apprentice Homework Assignment | HTTP library

Your Kafka and Zookeeper deployments are running in the kaf namespace according to your screenshots, presumably you have set this up manually and applied the configurations while in that namespace? Neither the Kafka or Zookeeper YAML files explicitly state a namespace in metadata, so will be deployed to the active namespace when created.

Anyway, the Kafka deployment YAML you have is hardcoded to assume Zookeeper is setup in the default namespace, with the following line:

Only a broad web crawl covering many web sites (hosts / domains) will profit from adding more Hadoop nodes. If only a small number of sites is crawled, parallelization will not make Nutch faster. Nutch configured to behave polite by default and does not access a single site in parallel and also waits between successive fetches from the same site.

But there are ways to make Nutch crawl a single site faster.

1. to make a single fetcher task faster (and more aggressively fetching from a single host (or domain depending on the value of partition.url.mode) the following configuration properties need to be adapted: fetcher.server.delay, fetcher.threads.per.queue and maybe other fetcher properties.

2. to allow more fetcher tasks (Hadoop nodes) crawl the same web site in parallel, URLPartitioner's getPartition method needs to be modified, see this discussion.

Be aware that making Nutch more aggressive without consent will probably result in complaints by the admins of the crawled web sites and increases the likelihood to get blocked!

In case anyone comes to this error. Even configuring the proxy at server and container level didn't worked. I'd found out that Jenkins has a proxy configuration at the application level. So I went into Jenkins, Administration section and configured the proxy properly. After that was done, it all started to work.

So this turned out to be an issue with a Producer sending messages to Kafka in a US date format rather than UK. Therefore, it created messages that would appear to be timestamped in the future - hence not be older than 100 hours and eligible for deletion.

To remove the existing message we set log.retention.bytes which prunes messages irrespective of the log.retention.hours setting. This caused the kafka topic to be pruned and delete the erroneous message - we then unset log.retention.bytes.

Next we set the log.message.timestamp.type=LogAppendTime to ensure that messages are stamped with a queue time as apposed to the document time. This will prevent bad dates from producers causing this issue again in the future.

If data goes to one broker, then the other, that depends on your partition count and excludes any connection issues

Still wonder is it a cluster now ? , however both the nodes have their own zookeepers

First, as mentioned in the documentation, zookeeper.connect needs to be the same string for a cluster to be formed. You only need one Zookeeper server, so stop the one on 2180 and just use localhost:2181

Once both Kafka brokers are running, you can use kafkacat -L or in Java AdminClient.describeCluster to verify the cluster metadata

After reading this aspnetcoreissue discovered that the problem was with the implementation of my IHostedService implementation that makes the request to Kafka.

The StartAsync method was performing the task, running until the request completed. By design this method is meant to be fire and forget, i.e. start the task and then continue. Updated my KafkaAdmin service to be a BackgroundService, overriding ExecuteAsync method, as listed below. Subsequently, tests no longer blocks.

After going through several articles on MVVM + Clean Code, I came to a conclusion that I cannot be using any dependencies related to android framework inside either domain or data module. Otherwise it will be violating the Dependency Inversion principle of SOLID.

Dependency Inversion principle

Higher level module should not be dependent on Lower level modules, and both should be dependent on abstraction.

In English -> You cannot directly access framework related components like database, gps, retrofit, etcetera from data or domain layers. They should not care about those stuffs. They should be totally independent of android related components. We can use interface to satisfy the rule of abstraction.

Therefore, my data module and domain module contains only language dependencies. Whatever android-framework-related data I want, I acquire it by implementing interfaces.

When describing topics using this tool, you only see the configurations that have been overriden. All other configurations, where the broker default is applied, are not listed.

You can override configuration at the topic level when:

• creating it: for example kafka-topics.sh -create --zookeeper localhost:2181 --replication-factor 3 --partitions 100 --topic test --config retention.bytes=12345

• altering it: for example kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name test --alter --add-config retention.bytes=12345

In these cases, when describing the topic, you will see its configs:

I got it working by looking at the cURL request with the Google API Explorer. The access token parameter was missing: