fluentular | : pencil : Fluentular is a Fluentd regular expression editor

I have a basic nginx deployment serving static content running on a GKE cluster. I have configured Stackdriver Logging for the cluster as per instructions here (I enabled logging for an existing cluster), and I also enabled the Stackdriver Kubernetes Monitoring feature explained here. The logging itself seems to be working fine, as I can see the logs from nginx in Stackdriver.

I am trying to create some log-based metrics like number of fulfilled 2xx requests, but all I am getting in the log entries in Stackdriver is the textPayload field. From what I understand, enabling Stackdriver Monitoring on the cluster spins up some Fluentd agents (which I can see if I run kubectl get pods -n kube-system), and they should have an nginx log parser enabled by default (as per documentation here). However, none of the log entries that show up in Stackdriver have the jsonPayload field that should be there for structured logs.

I'm using the default log_format config for nginx, and I've verified that the default nginx parser is able to parse the logs my application is writing (I copied the default Fluentd nginx parser plugin regular expression and a log entry from my application to this tool and it was able to parse the entry)

I'm sure I must be missing something, but I can't figure out what.

Edit:

For reference, here is my NGINX log format:

I am trying to parse the logs from all the OpenStack services and send it to S3 in JSON.

I am able to get parse the logs with this multiline format.

I am trying to parse daemon logs from my linux machine to elastic search using fluentd but having hard time creating regex pattern for it. Below are few of the logs from the daemon logs:

I am trying to use the regex filter to parse my log My regex expression and sample string are as follows Regex: