[logstash-users] Match on CSV?

[Nathan Seery]

I'm getting logs from my firewalls (Palo Alto) and I'm having trouble
matching them.
I have a CSV parse setup and that works fine, but I'm not sure if I can
match on a csv.
The trouble is that some of the fields can be empty, but aren't always.

2015-05-29T09:12:38-05:00 TXCO-PA200 1,2015/05/29
09:12:37,001606011545,TRAFFIC,end,1,2015/05/29
09:12:37,10.7.16.21,8.8.8.8,96.88.0.137,8.8.8.8,Inside-to-Outside,,,dns,vsys1,txco-wifi-guest,Outside,ethernet1/4.706,ethernet1/1,Log
to TXCO-RASPI,2015/05/29
09:12:37,9599,1,62512,53,17676,53,0x400019,udp,allow,225,83,142,2,2015/05/29
09:12:07,0,any,0,56210658,0x0,10.0.0.0-10.255.255.255,US,0,1,1,aged-out
2015-05-29T10:12:26-05:00 ARLA-PA200 1,2015/05/29
10:12:25,001606017021,TRAFFIC,end,1,2015/05/29
10:12:25,10.3.3.10,132.245.15.226,50.76.217.137,132.245.15.226,Any-To-Outside,,,outlook-web-online,vsys1,Inside-Wired,Outside,ethernet1/2,ethernet1/4,Syslog
to RasPi,2015/05/29
10:12:25,51430,1,60534,443,47455,443,0x400053,tcp,allow,15551,1952,13599,25,2015/05/29
10:11:55,0,not-resolved,0,60150675,0x0,10.0.0.0-10.255.255.255,US,0,11,14,tcp-rst-from-client
2015-05-29T10:13:59-05:00 OKLA-PA200 1,2015/05/29
10:13:58,001606016734,TRAFFIC,end,1,2015/05/29
10:13:58,172.16.1.50,174.76.226.37,24.249.98.103,174.76.226.37,Any-To-Outside,,,web-browsing,vsys1,Inside-Wireless,Outside,ethernet1/3,ethernet1/1,OKLA-RasPi,2015/05/29
10:13:58,16780,1,63410,80,23690,80,0x40001c,tcp,allow,20110,2507,17603,33,2015/05/29
10:11:11,138,any,0,56632136,0x0,172.16.0.0-172.31.255.255,US,0,15,18
2015-05-29T10:14:45-05:00 SWUC-PA200 : 1,2015/05/29
10:14:45,001606011589,TRAFFIC,end,1,2015/05/29
10:14:45,172.22.1.105,199.59.150.11,66.196.230.84,199.59.150.11,Any-to-Outside,,,twitter-base,vsys1,Inside-Wireless,Outside,ethernet1/4.205,ethernet1/1,SysLog,2015/05/29
10:14:45,19405,1,59169,443,12122,443,0x404053,tcp,allow,46857,6422,40435,94,2015/05/29
10:08:59,331,not-resolved,0,339496888,0x0,172.16.0.0-172.31.255.255,US,0,44,50,tcp-fin
2015-05-29T10:15:08-05:00 SWRG-PA200 1,2015/05/29
10:15:07,001606020851,TRAFFIC,end,1,2015/05/29
10:15:07,10.5.2.17,104.69.227.233,66.196.247.82,104.69.227.233,Any-To-Outside,,,ms-update,vsys1,Inside-Wired,Outside,ethernet1/2,ethernet1/4,swrg-raspi,2015/05/29
10:15:07,23173,1,62740,80,12853,80,0x40001c,tcp,allow,5508,803,4705,15,2015/05/29
10:13:53,60,not-resolved,0,147291919,0x0,10.0.0.0-10.255.255.255,US,0,8,7,tcp-fin
2015-05-29T10:15:23-05:00 LSC-PA500 1,2015/05/29
10:15:23,009401002674,TRAFFIC,end,1,2015/05/29
10:15:23,10.15.5.2,10.15.5.1,0.0.0.0,0.0.0.0,Server-to-Server,,,dns,vsys1,Inside-Server,Inside-Server,ethernet1/2.4,ethernet1/2.4,LSC-SRV,2015/05/29
10:15:23,30086,1,43008,53,0,0,0x19,udp,allow,259,86,173,2,2015/05/29
10:14:51,0,any,0,49585888,0x0,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,1,1,aged-out

Descriptions of all the fields are found here:
https://www.paloaltonetworks.com/documentation/60/pan-os/pan-os/reports-and-logging/syslog-field-descriptions.html

csv {
columns => ["FUTURE_USE", "Receive Time", "Serial
Number", "Type", "Subtype", "FUTURE_USE", "Generated Time", "Source IP",
"Destination IP", "NAT Source IP", "NAT
Destination IP", "Rule Name", "Source User", "Destination User",
"Application",
"Virtual System", "Source Zone",
"Destination Zone", "Ingress Interface", "Egress Interface", "Log
Forwarding Profile",
"FUTURE_USE", "Session ID", "Repeat
Count", "Source Port", "Destination Port", "NAT Source Port", "NAT
Destination Port",
"Flags", "Protocol", "Action", "Bytes",
"Bytes Sent", "Bytes Received", "Packets", "Start Time", "Elapsed Time",
"Category",
"FUTURE_USE", "Sequence Number", "Action
Flags", "Source Location", "Destination Location", "FUTURE_USE", "Packets
Sent",
"Packets Received", "Session End Reason"]
}

The CSV works fine one the one FW that is local and easy to define 'type'.

if [type] == "paloalto"

Because the other FWs aren't local, they're lumped in with all the other
log files from their respective remote offices.

Using the Grok Debugger <http://grokconstructor.appspot.com> I can make
some progress with the above log lines, but I'm not sure how to handle the
potentially empty columns.

Is it possible to simply put the CSV in the match?
If yes, how does that work?
If no, what is the recommended method?

Thanks in advance!