multiline

The multiline codec will collapse multiline messages and merge them into a single event.

The original goal of this codec was to allow joining of multiline messages from files into a single event. For example, joining Java exception and stacktrace messages into a single event.

The config looks like this:

input {
  stdin {
    codec => multiline {
      pattern => "pattern, a regexp"
      negate => true or false
      what => "previous" or "next"
    }
  }
}

The pattern should match what you believe to be an indicator that the field is part of a multi-line event.

The what must be previous or next and indicates the relation to the multi-line event.

The negate can be true or false (defaults to false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)

For example, Java stack traces are multiline and usually have the message starting at the far-left, with each subsequent line indented. Do this:

input {
  stdin {
    codec => multiline {
      pattern => "^\\s"
      what => "previous"
    }
  }
}

This says that any line starting with whitespace belongs to the previous line.

Another example is to merge lines not starting with a date up to the previous line..

input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      # Grok pattern names are valid! :)
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => "previous"
    }
  }
}

This says that any line not starting with a timestamp should be merged with the previous line.

One more common example is C line continuations (backslash). Here’s how to do that:

filter {
  multiline {
    pattern => "\\$"
    what => "next"
  }
}

This says that any line ending with a backslash should be combined with the following line.

Decoder usage

Synopsys

codec => multiline {
    # The regular expression to match with the line
    pattern => "^\\s"

}

Available settings

Setting Type Info Default value
charset string "UTF-8"
delimiter string "\n"
negate bool false
pattern string required ""
what string previous or next "previous"

Details

charset

  • Value type is string
  • Default value is "UTF-8"
  • One of "US-ASCII" "ISO-8859-1" "latin1" "arabic" "hebrew" "Shift_JIS" "EUC-KR" "ISO-2022-JP" "ISO-8859-15" "IBM869" "windows-1252" ...

    See the IANA registry for more details

    delimiter

  • Value type is string
  • Default value is "\n"
  • Change the delimiter that separates lines

    negate

  • Value type is bool
  • Default value is false
  • Negate the regexp pattern (if not matched).

    pattern

  • This is a required setting
  • Value type is string
  • Default value is ""
  • The regular expression to match with the line

    what

  • Value type is string
  • Possible values : previous or next
  • Default value is "previous"
  • If the pattern matched, does event belong to the next or previous event?

    Full configuration blueprint

    Expand me...