syslog | Golang alternative for built-in log/syslog package

We are overriding console.log in our productive ExpressJS application to add certain fields (e.g. timestamp, sessionid, requestid) as well as send logs to a syslog server.

We are doing this by adding console.requestId and console.sessionId properties to the global console object and then overriding console.log() to output these values as well as send logs to syslog. This works fine:

I'm learning that Golang channels are actually slower than many alternatives provided by the language. Of course, they are really easy to grasp but because they are a high level structure, they come with some overhead.

Reading some articles about it, I found someone benchmarking the channels here. He basically says that the channels can transfer 10 MB/s, which of course must be dependant on his hardware. He then says something that I haven't completely understood:

If you just want to move data quickly using channels then moving it 1 byte at a time is not sensible. What you really do with a channel is move ownership of the data, in which case the data rate can be effectively infinite, depending on the size of data block you transfer.

I've seen this "move ownership of data" in several places but I haven't seen a solid example illustrating how to do it instead of moving the data itself.

I wanted to see an example in order to understand this best practice.

I have a docker-compose setup with containers logging into fluentd. To support different demo environments, I have events being output to multiple destinations (ElasticSearch, Splunk, Syslog, etc.)

I would like to maintain a single configuration file, but disable output plugins that are not needed. If I have 4 potential output destinations, I would have to maintain 10 different configuration files to support all the different possible combinations.

I know that plugins can use environment variables for configuration parameters, which would be ideal. However, I don't see that there is a common 'enabled' or 'disable' parameter in the underlying plugin architecture.

Is there any way to disable a plugin externally? Or will I have to dynamically build my configuration file from an external script?

I have one table which contains the data and all column are in text type.

Sample data as shown below

I have a azure data-explor that queries some syslogs coming-in, filters and aggregate them. The output of this query is store on my local computer in a csv file. So every time a run my Python SDK, it runs a query and saves the output in a csv file.

What I am looking for, is to push that result of the query to a cosmosdb.

Looking into azure GitHub azure-sdk-for-python, I found a library that can achieve this result with this code.

I hope everyone is doing great.

I do have a question about azure data explorer and logstash.

I followed this Microsoft tutorial to configure a azure data explorer with Kusto and logstash.

https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-logstash

I created a azure data explorer, created a table for the data. on the other side I pinned up a virtual machine, installed logstash and configured all the minimum required to forward syslogs to azure data explorer.

Everything works just fine and I don't have any problem, but there is one step that is bugging me a bit and would like some clarifications.

I wanted to test this project with real time syslogs coming from my virtual machine.

so I configured conf file for logstash as follow:

I want to store my samba binary path to the global $PATH variable on my rpi4 but it did not work as expected. I've created a file samba-binary-path.sh in the folder /etc/profile.sh and made it with chmod +x executable.

The file samba-binary-path.sh contains the following:

export PATH=/usr/local/samba/bin/:/usr/local/samba/sbin/:$PATH

Furthermore I have export PATH=/usr/local/samba/bin/:/usr/local/samba/sbin/:$PATH also saved in the file /etc/environment.

Now comes the crazy part. When I execute my script on my cli it work as intended but when it get started from another process the path variable missing my samba binary path.

The affected code block:

I have below code which I am running on linux. It is throwing lot of errors even though the json formatter is correct.

I use my custom fluentd plugin and it does not work with ubuntu20 but for other ubuntu version is no problem.

Here is my error