Discussion:
[logstash-users] dynamically loading grok rules
Naveen Rohatgi
2015-05-20 21:40:40 UTC
Permalink
Hello,
I have a requirement to dynamically load the grok rules from a .yml file when an environment variable indicates a change in the rules file. I am planning to run log stash with a single filter thread, so the new filter may not be thread safe. I have tried but I keep getting the following error message when I call grok.register. I am currently testing with a single grok pattern in the 3 line yml file.

The error reported is:
can't modify frozen string
org/jruby/RubyString.java:2763:in `sub!'
/home/seceon/logstash-1.4.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.10.12/lib/grok-pure.rb:124:in `compile'
org/jruby/RubyKernel.java:1521:in `loop'
/home/seceon/logstash-1.4.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.10.12/lib/grok-pure.rb:91:in `compile'
/home/seceon/logstash-1.4.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.10.12/lib/grok/pure/pile.rb:51:in `compile'
/home/seceon/logstash-1.4.0/lib/logstash/filters/grok.rb:277:in `register'
org/jruby/RubyArray.java:1613:in `each'
/home/seceon/logstash-1.4.0/lib/logstash/filters/grok.rb:275:in `register'
org/jruby/RubyHash.java:1339:in `each'
/home/seceon/logstash-1.4.0/lib/logstash/filters/grok.rb:266:in `register'
/home/seceon/logstash-1.4.0/logstash/filters/translate.rb:127:in `register'
org/jruby/RubyArray.java:1613:in `each'

The .yml file contents are
{
"Bad protocol version identification %{DATA:version} from %{IP:src_ip}": {
"classification": "suspect"
}
}

My code snippet is
....
@grok_filter = LogStash::Filters::Grok.new(
"pattern" => dictionary.keys
)
....

However, if I put the same string in the code, it works as following:
@grok_filter = LogStash::Filters::Grok.new(
"pattern" => "Bad protocol version identification %{DATA:version} from %{IP:src_ip}"
)

I understand I am trying to modify something immutable, but how to avoid it. Please help me. I am new to ruby & log stash.

Thanks,
Naveen
--
Remember: if a new user has a bad time, it's a bug in logstash.
---
You received this message because you are subscribed to the Google Groups "logstash-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to logstash-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Magnus Bäck
2015-05-25 06:04:57 UTC
Permalink
On Wednesday, May 20, 2015 at 23:40 CEST,
Post by Naveen Rohatgi
I have a requirement to dynamically load the grok rules from
a .yml file when an environment variable indicates a change
in the rules file.
How would the environment variable indicate this? You can't
affect the Logstash process's environment from the outside.
Post by Naveen Rohatgi
I am planning to run log stash with a single filter thread, so
the new filter may not be thread safe. I have tried but I keep
getting the following error message when I call grok.register.
I am currently testing with a single grok pattern in the 3 line
yml file.
can't modify frozen string
org/jruby/RubyString.java:2763:in `sub!'
/home/seceon/logstash-1.4.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.10.12/lib/grok-pure.rb:124:in `compile'
[...]
Post by Naveen Rohatgi
My code snippet is
....
@grok_filter = LogStash::Filters::Grok.new(
"pattern" => dictionary.keys
)
....
Wild guess: Use dictionary.keys.dup to create a copy of the list of
dictionary keys.

[...]
--
Magnus Bäck | Software Engineer, Development Tools
***@sonymobile.com | Sony Mobile Communications
--
Remember: if a new user has a bad time, it's a bug in logstash.
---
You received this message because you are subscribed to the Google Groups "logstash-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to logstash-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
a***@logz.io
2015-05-28 13:58:47 UTC
Permalink
I don't think it's possible to update Logstash grok rules especially since
Logstash has internal queues.

We have implemented this feature by having a system that whenever the
configuration changes it spins up new Logstash clients and we change the
routing in our DNS to the new instances and then let the old ones drain and
shut them down. It works very nicely and we update configuration all the
time.

-- ay
Post by Magnus Bäck
On Wednesday, May 20, 2015 at 23:40 CEST,
Post by Naveen Rohatgi
I have a requirement to dynamically load the grok rules from
a .yml file when an environment variable indicates a change
in the rules file.
How would the environment variable indicate this? You can't
affect the Logstash process's environment from the outside.
Post by Naveen Rohatgi
I am planning to run log stash with a single filter thread, so
the new filter may not be thread safe. I have tried but I keep
getting the following error message when I call grok.register.
I am currently testing with a single grok pattern in the 3 line
yml file.
can't modify frozen string
org/jruby/RubyString.java:2763:in `sub!'
/home/seceon/logstash-1.4.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.10.12/lib/grok-pure.rb:124:in
`compile'
[...]
Post by Naveen Rohatgi
My code snippet is
....
@grok_filter = LogStash::Filters::Grok.new(
"pattern" => dictionary.keys
)
....
Wild guess: Use dictionary.keys.dup to create a copy of the list of
dictionary keys.
[...]
--
Magnus BÀck | Software Engineer, Development Tools
--
Remember: if a new user has a bad time, it's a bug in logstash.
---
You received this message because you are subscribed to the Google Groups "logstash-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to logstash-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...