Zeek | Python distributed web scrapper and dynamic crawler | Crawler library

In distributed mode, would I still want to submit just a single elasticsearch.properties through a single API call?

It'd be a JSON file, but yes.

what dictates the number of workers?

Up to you. JVM usage is one factor that you can monitor and scale on

Not really any documentation that I am aware of

A common problem when running traffic through Zeek is that packets may have invalid checksums. Zeek by default skips such packets, so the net result is missing logs/files/artifacts that the user expects to be there. Often those invalid checksums are caused by checksum offloading, where the packet capture process grabs transmitted packets before the NIC had a chance to fix the checksums.

Zeek normally warns when it encounters invalid checksums -- look for the something resembling the following on stderr, or in reporter.log:

Your trace file likely has invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Zeek unless using the -C command-line option or toggling the 'ignore_checksums' variable.

(This is from find-checksum-offloading.zeek, which is included in Zeek's default configuration.)

You have many options here. You can:

• run Zeek with -C, as per the above
• say redef ignore_checksums=T; in a script (usually local.zeek)
• add the redef at the command line: zeek -r the.pcap ... ignore_checksums=T
• fix the checksums in the pcap, e.g. with tcprewrite -C -i input.pcap -o fixed.pcap (tcprewrite ships with tcpreplay) -- this is best if others will consume your pcap too.

You can check the exact error message in the journal logs, as it is suggested by systemctl:

See "systemctl status syslog-ng.service" and "journalctl -xe" for details.

Alternatively, you can start syslog-ng in the foreground:

$ syslog-ng -F --stderr

You probably have a persist-name collision due to the matching elasticsearch-http() URLs. Please try adding the persist-name() option with 2 unique names, for example:

So your problem is that you are mixing up the diffrence between bot and client. Those are to diffrent things.
A bot is simpler, it just recivies commands from you and handles them. Meaning it does something. If you want to write in the chat and do more things you need a client.

Besides that you can't create a Bot like that: client = commands.Bot(command_prefix=bot_prefix) The right way would be: bot = Bot(command_prefix='$') since you have already imported bot.

The way to go for you would be to stick with the client and instead of commands use:

Seth Hall nailed it. I messed up the rules without knowing. Thankfully an easy fix. Thanks.

The variable is defined (with an empty string value) as a redef'able constant in the init-bare.zeek file that comes with the distribution. So just say

I was experiencing the same error for my standalone setup. Found this question from googling it. More googling the error brought me to a few blogs including one in which the comments mentioned the same error. The author mentioned giving the binaries permissions using setcap:

Not out of the box, no. One way to add it would be to expand the TCP_Flags class in your local build so it captures the TCP header's th_x2 field bits as well. Then, use the tcp_packet event, which reports the flags.

This would be quite slow, though, as it'd be packet-level analysis.