Несколько шаблонов для анализа в Logstash

В моем файле журнала есть несколько шаблонов, включая журналы в формате JSON. Я хочу проанализировать несколько шаблонов в плагине Grok, но, похоже, он не работает.

'filter {grok {  break_on_match => false 
match =>[ "message", "%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} (?<threadName>[^:]+):%{NUMBER:ThreadID} - %{GREEDYDATA:Line}",
           "message","%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} (?<threadName>[^:]+):%{NUMBER:ThreadID} - %{IP:Clicnet} - - %{GREEDYDATA:Line}"]}
           json {source => "Line"}mutate{remove_field => [ "Line","ThreadID" ]}}'

Несмотря на то, что строка с строкой JSON успешно проанализирована, есть тег grokparsefailure.

2017-01-27 11:54:48 INFO PropertiesReader: 33 - {"timestamp": 1485518878968, "h": "297268184dde", "l": "INFO", "cN": "org.com. logstash.demo "," mN ":" loadProperties "," m ":" загрузить файл свойств из / var / tmp / conf "}

{
       "message" => "2017-01-27 11:54:48 INFO PropertiesReader:33 - {\"timestamp\":1485518878968,\"h\":\"297268184dde\", \"l\":\"INFO\", \"cN\":\"org.com.logstash.demo\", \"mN\":\"loadProperties\", \"m\":\"load property file from /var/tmp/conf\"}",
      "@version" => "1",
    "@timestamp" => "2017-03-20T17:19:16.316Z",
          "type" => "stdin",
          "host" => "ef3b82",
       "LogDate" => "2017-01-27 11:54:48",
      "loglevel" => "INFO",
    "threadName" => "PropertiesReader",
          "tags" => [
        [0] "_grokparsefailure"
    ],
     "timestamp" => 1485518878968,
             "h" => "297268184dde",
             "l" => "INFO",
            "cN" => "org.com.logstash.demo",
            "mN" => "loadProperties",
             "m" => "load property file from /var/tmp/conf"
}

и вторая строка, в которой нет JSON, полностью не работает

2017-01-20 15:46:16 INFO RequestLog: 60 - 10.252.134.34 - - [20 / янв / 2017: 15: 46: 16 +0000] "ОПЦИИ //127.0.0.0:8080/ HTTP / 1.1 "404 237 1

Error parsing json {:source=>"Line", :raw=>["10.252.134.34 - - [20/Jan/2017:15:46:16 +0000] \"OPTIONS //127.0.0.0:8080/ HTTP/1.1\" 404 237  1", "[20/Jan/2017:15:46:16 +0000] \"OPTIONS //127.0.0.0:8080/ HTTP/1.1\" 404 237  1"], :exception=>java.lang.ClassCastException: org.jruby.RubyArray cannot be cast to org.jruby.RubyIO, :level=>:warn}
{
       "message" => "2017-01-20 15:46:16 INFO  RequestLog:60 - 10.252.134.34 - - [20/Jan/2017:15:46:16 +0000] \"OPTIONS //127.0.0.0:8080/ HTTP/1.1\" 404 237  1",
      "@version" => "1",
    "@timestamp" => "2017-03-20T17:19:51.175Z",
          "type" => "stdin",
          "host" => "ef3b82",
       "LogDate" => [
        [0] "2017-01-20 15:46:16",
        [1] "2017-01-20 15:46:16"
    ],
      "loglevel" => [
        [0] "INFO",
        [1] "INFO"
    ],
    "threadName" => [
        [0] " RequestLog",
        [1] " RequestLog"
    ],
       "Clicnet" => "10.252.134.34",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}

elastic-stack logstash logstash-grok
person BobCoder    schedule 20.03.2017    source источник

Ответы (1)


arrow_upward
1
arrow_downward

Потратив 5 часов, мне удалось найти решение. Используемый ниже шаблон, который успешно проанализировал обе строки журнала

/opt/logstash/bin/logstash -e 'filter {grok  { match =>{ "message" =>["%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} (?<threadName>[^:]+):%{NUMBER:ThreadName} - %{IP:Client} - - %{GREEDYDATA:LogMessage}", "%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} (?<threadName>[^:]+):%{NUMBER:ThreadID} - %{GREEDYDATA:Line}"]}}  json {source => "Line"} mutate{remove_field => [ "Line","ThreadID" ]}}'
person BobCoder    schedule 20.03.2017