[logstash-users] Using grok for a single file with multiple patterns

[Paul Chauvet]

Hi all,

I'm kinda stuck on trying to do something with logstash. I'm trying to use
it to consume and parse files from the Zimbra mail server right now -
starting with the audit.log file. That file unfortunately has several
types of information (IMAP, Web Client, Caldav, ActiveSync, etc.). To
handle this, I setup multiple patterns I get a grokparsefailure on all of
these (though they were tested in the GrokDebugger).

My config on the logstash server is as follows:

filter {
if [type] == "zimbra_audit" {
grok {
break_on_match => false
patterns_dir => [ "/etc/logstash/patterns.d" ]
match => [
"message", "%{ZIMBRA_AUDIT_IMAPPOP}",
"message", "%{ZIMBRA_AUDIT_WEB}",
"message", "%{ZIMBRA_AUDIT_WEBFAIL}",
"message", "%{ZIMBRA_AUDIT_ACTIVESYNC}",
"message", "%{ZIMBRA_AUDIT_CALDAV}"
]
}
}
}

I'm not sure that what I'm doing (listing multiple patterns, each as a
separate message) is done correctly or even possible. Any advice you can
provide on this would be appreciated!

P.S. I've included some of those below from a file in my patterns
directory (I'll neaten these up eventually). The first is a preface that
is the same on each line, the next two are patterns that differ in the log.

ZIMBRA_AUDIT_PREFACE %{TIMESTAMP_ISO8601:Timestamp} %{LOGLEVEL:LogLevel}
+%{SYSLOG5424SD:Connection}
ZIMBRA_AUDIT_IMAPPOP %{ZIMBRA_AUDIT_PREFACE}
\[ip=%{IPV4:Proxy_IP};oip=%{IPV4:Client_IP};\] security - cmd=Auth;
account=%{GREEDYDATA:Account}; protocol=%{WORD:Protocol};
ZIMBRA_AUDIT_WEB %{ZIMBRA_AUDIT_PREFACE}
\[name=%{USERNAME:user}@%{JAVACLASS:Domain};oip=%{IPV4:Client_IP};ua=zclient/7.2.7_GA_2942;\]
security - cmd=Auth; account=%{GREEDYDATA:Account};
protocol=%{WORD:Protocol};


Thanks,

Paul Chauvet

[Jon Gerdes]

Some sample logs would be nice to start with 8)

[Paul Chauvet]

Sorry! I didn't paste logs initially since I assumed that the issue wasn't
my parsing but the setup of the configuration file. Here's a couple
examples with IP addresses and user info altered. All log lines I've tried
have worked fine against grokdebug.herokuapp.com (after some trial and
error of course).

2014-10-24 13:21:11,766 INFO [ImapServer-22235] [ip=127.0.0.1;oip=
192.168.0.1;] security - cmd=Auth; account=***@ourserver.domain.edu
<http://zmail.newpaltz.edu/>; protocol=imap;

2014-10-24 13:22:35,041 INFO
[btpool0-55634://localhost/service/soap/AuthRequest]

Post by Jon Gerdes
Some sample logs would be nice to start with 8)

[xianzhi li]

Same issue here, and i am using the latest logstash 1.4.2. Looks like grok
only takes one match pattern in this case.

Post by Paul Chauvet
Hi all,
I'm kinda stuck on trying to do something with logstash. I'm trying to
use it to consume and parse files from the Zimbra mail server right now -
starting with the audit.log file. That file unfortunately has several
types of information (IMAP, Web Client, Caldav, ActiveSync, etc.). To
handle this, I setup multiple patterns I get a grokparsefailure on all of
these (though they were tested in the GrokDebugger).
filter {
if [type] == "zimbra_audit" {
grok {
break_on_match => false
patterns_dir => [ "/etc/logstash/patterns.d" ]
match => [
"message", "%{ZIMBRA_AUDIT_IMAPPOP}",
"message", "%{ZIMBRA_AUDIT_WEB}",
"message", "%{ZIMBRA_AUDIT_WEBFAIL}",
"message", "%{ZIMBRA_AUDIT_ACTIVESYNC}",
"message", "%{ZIMBRA_AUDIT_CALDAV}"
]
}
}
}
I'm not sure that what I'm doing (listing multiple patterns, each as a
separate message) is done correctly or even possible. Any advice you can
provide on this would be appreciated!
P.S. I've included some of those below from a file in my patterns
directory (I'll neaten these up eventually). The first is a preface that
is the same on each line, the next two are patterns that differ in the log.
ZIMBRA_AUDIT_PREFACE %{TIMESTAMP_ISO8601:Timestamp} %{LOGLEVEL:LogLevel}
+%{SYSLOG5424SD:Connection}
ZIMBRA_AUDIT_IMAPPOP %{ZIMBRA_AUDIT_PREFACE}
\[ip=%{IPV4:Proxy_IP};oip=%{IPV4:Client_IP};\] security - cmd=Auth;
account=%{GREEDYDATA:Account}; protocol=%{WORD:Protocol};
ZIMBRA_AUDIT_WEB %{ZIMBRA_AUDIT_PREFACE}
security - cmd=Auth; account=%{GREEDYDATA:Account};
protocol=%{WORD:Protocol};
Thanks,
Paul Chauvet

[Jon Gerdes]

Grok can work on multiple matches OK - at least in 1.4.2. I have one
running which definatly works. Now it comes down to a matter of taste.
You can use a single grok with multiple matches but you can't tell easily
which one actually did the job. Also Paul, you have what look like five
specific match patterns but have set break_on_match to false - why is that?
Ahh just re read your post.

I suggest breaking this up into two groks - the first one to break out the
general info, that one could overwrite message to only contain the
remainder and have a tag_on_failure to indicate failure of that for
testing. Here's an example for a generic syslog entry:

grok {
tag_on_failure => [ "BROKEN_GROK_SYSLOG", "_grokparsefailure" ]
overwrite => [ "message" ]
match => [ "message", "%{SYSLOGBASE2}
%{GREEDYDATA:message}" ]
}

Then follow that with another grok with the multiple matches but you can
now remove the common fields making for shorter matches. They very rapidly
disappear off the right margin!

Another way to do it is with mutiple if [message] =~ /regex/ { } style
conditionals and then operate on each message type individually, say start
with a grok and then mutates and geoip etc.

Cheers
Jon

[Paul Chauvet]

Thanks for the reply Jon. I thought what I was originally trying
I'm not 100% clear but I think what you're suggesting is something like
the following:

filter {
if [type] == "zimbra_audit" {
grok {
break_on_match => false
patterns_dir => [ "/etc/logstash/patterns.d" ]
overwrite => ["message"]
match => ["message", "%{ZIMBRA_AUDIT_PREFACE} %{GREEDYDATA:message}"]
}

grok {
match => {
["message", "%{ZIMBRA_AUDIT_IMAPPOP}",
"message", "%{ZIMBRA_AUDIT_WEB}",
"message", "%{ZIMBRA_AUDIT_WEBFAIL}",
"message", "%{ZIMBRA_AUDIT_ACTIVESYNC}",
"message", "%{ZIMBRA_AUDIT_CALDAV}"]
}
}
}


The first section has the info that is the same on each line, the second
second has the differences. With these changes though, Logstash is still
just clumping everything from this system as "message" with
_grokparsefailure set.