[logstash-users] File output filter writing logs out of order

[b***@logicmonitor.com]

We are using pretty trivial logstash setup. Sender configs look something
like


input {

file {
path => "/var/log/secure"
type => "secure"
}
}

output {

tcp {
host => "log02.us-east-1.logicmonitor.net"
port => 2009
mode => "client"
codec => "json"
workers => 10
}

}

And the receiver is

input {

tcp {
mode => "server"
port => 2009
codec => "json"
}

}

output {

file {
path => "/log/%{new-host}/%{path}"
codec => "json"
message_format => "%{message}"
workers => 30
}

}


We are finding that logs entries are being written out of order (seemingly
random) to the destination files on the receiver. This problem seems to
increase as the number of logs being sent to our receivers increases.
Admittedly, we are using an older 1.2 version of logstash and my first
step is to upgrade to the latest. However, I wanted to check to see if
there is something known going on.

Thanks.

[Magnus Bäck]

On Monday, June 01, 2015 at 16:38 CEST,

Post by b***@logicmonitor.com
We are using pretty trivial logstash setup. Sender configs look
something like

[...]

Post by b***@logicmonitor.com
output {
tcp {
host => "log02.us-east-1.logicmonitor.net"
port => 2009
mode => "client"
codec => "json"
workers => 10
}
}

[...]

Post by b***@logicmonitor.com
output {
file {
path => "/log/%{new-host}/%{path}"
codec => "json"
message_format => "%{message}"
workers => 30
}
}
We are finding that logs entries are being written out of order
(seemingly random) to the destination files on the receiver.

With multiple output workers for both the tcp and file outputs that's
expected. There are no ordering guarantees.

This mailing list is deprecated. Prefer https://discuss.elastic.co/
in the future.

[...]
--
Magnus Bäck | Software Engineer, Development Tools
***@sonymobile.com | Sony Mobile Communications