nishang | red team, penetration testing and offensive security | Security Testing library

Grok is a wrapper around regular expressions. If you can parse data with a regex, you can implement it with grok.

Even though your scope is specific to the CommandLine field, parsing each of the fields in most key=value logs is pretty straightforward, and a single regex can be used for every field with some grok filters. If you intend to store, query, and visualize logs - the more data, the better.

Regular Expression:

First we start with the following:

(.*?(?=\s\w+=|\<|$))

• .*? - Matches any character except for line terminators
• (?=\s\w+=|\<|$)) - Positive lookahead that asserts the pattern must match the following
  • \s\w+= - Any word characters with a space prior to it, followed by a =
  • |\<|$ - Alternatively may match < or the end of the line so as not to include them in the matching group.

This means that each field can be parsed similar to the following:

CommandLine=(.*?(?=\s\w+=|\<|$))

Grok:

Now this means we can begin creating grok filters. The power of it is that reusable components may have semantic language applied to them.

/etc/logstash/patterns/powershell.grok:

# Patterns
PS_KEYVALUE (.*?(?=\s\w+=|\<|$))

# Fields
PS_NEWCOMMANDSTATE NewCommandState=%{PS_KEYVALUE:NewCommandState}
PS_SEQUENCENUMBER SequenceNumber=%{PS_KEYVALUE:SequenceNumber}
PS_HOSTNAME HostName=%{PS_KEYVALUE:HostName}
PS_HOSTVERSION HostVersion=%{PS_KEYVALUE:HostVersion}
PS_HOSTID HostId=%{PS_KEYVALUE:HostId}
PS_HOSTAPPLICATION HostApplication=%{PS_KEYVALUE:HostApplication}
PS_ENGINEVERSION EngineVersion=%{PS_KEYVALUE:EngineVersion}
PS_RUNSPACEID RunspaceId=%{PS_KEYVALUE:RunspaceId}
PS_PIPELINEID PipelineId=%{PS_KEYVALUE:PipelineId}
PS_COMMANDNAME CommandName=%{PS_KEYVALUE:CommandName}
PS_COMMANDTYPE CommandType=%{PS_KEYVALUE:CommandType}
PS_SCRIPTNAME ScriptName=%{PS_KEYVALUE:ScriptName}
PS_COMMANDPATH CommandPath=%{PS_KEYVALUE:CommandPath}
PS_COMMANDLINE CommandLine=%{PS_KEYVALUE:CommandLine}

Where %{PATTERN:label} will utilize the PS_KEYVALUE regular expression, and the matching group will be labeled with that value in JSON. This is where you can get flexible in naming fields you know.

/etc/logstash/conf.d/powershell.conf:

input {
    ...
}

filter {
    grok {
        patterns_dir => "/etc/logstash/patterns"
        break_on_match => false
        match => [
            "message", "%{PS_NEWCOMMANDSTATE}",
            "message", "%{PS_SEQUENCENUMBER}",
            "message", "%{PS_HOSTNAME}",
            "message", "%{PS_HOSTVERSION}",
            "message", "%{PS_HOSTID}",
            "message", "%{PS_HOSTAPPLICATION}",
            "message", "%{PS_ENGINEVERSION}",
            "message", "%{PS_RUNSPACEID}",
            "message", "%{PS_PIPELINEID}",
            "message", "%{PS_COMMANDNAME}",
            "message", "%{PS_COMMANDTYPE}",
            "message", "%{PS_SCRIPTNAME}",
            "message", "%{PS_COMMANDPATH}",
            "message", "%{PS_COMMANDLINE}"
        ]
    }
}

output {
    stdout { codec => "rubydebug" }
}

Result:

{
    "HostApplication" => "C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe",
      "EngineVersion" => "5.1.14409.1005",
         "RunspaceId" => "bd4224a9-ce42-43e3-b8bb-53a302c342c9",
            "message" => "<'Data'>NewCommandState=Stopped SequenceNumber=1463 HostName=ConsoleHost HostVersion=5.1.14409.1005 HostId=b99970c6-0f5f-4c76-9fb0-d5f7a8427a2a HostApplication=C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe EngineVersion=5.1.14409.1005 RunspaceId=bd4224a9-ce42-43e3-b8bb-53a302c342c9 PipelineId=167 CommandName=Import-Module CommandType=Cmdlet ScriptName= CommandPath= CommandLine=Import-Module -Verbose.\\nishang.psm1<'/Data'>",
             "HostId" => "b99970c6-0f5f-4c76-9fb0-d5f7a8427a2a",
        "HostVersion" => "5.1.14409.1005",
        "CommandLine" => "Import-Module -Verbose.\\nishang.psm1",
         "@timestamp" => 2017-05-12T23:49:24.130Z,
               "port" => 65134,
        "CommandType" => "Cmdlet",
           "@version" => "1",
               "host" => "10.0.2.2",
     "SequenceNumber" => "1463",
    "NewCommandState" => "Stopped",
         "PipelineId" => "167",
        "CommandName" => "Import-Module",
           "HostName" => "ConsoleHost"
}