/blog/2014/12/setting-up-logstash-1-4-2-to-forward-nginx-logs-to-elasticsearch/

[bravo-kernel]

Setting up Logstash 1.4.2 to forward Nginx logs to Elasticsearch

https://www.bravo-kernel.com/blog/2014/12/setting-up-logstash-1-4-2-to-forward-nginx-logs-to-elasticsearch/

[bravo-kernel]

Tarun Madiraju [DISQUS]: Does having two different patterns with the same name (as you have done with NGINXACCESS) work? I quickly tested it out, you would receive grok failures for that implementation.

1. Doesn't work

NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}  
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}

grok {  
  match => { "message" => "%{NGINXACCESS}" }  
}

2. To make it work, I guess would have to do something like this pattern

NGINXACCESSA %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}  
NGINXACCESSB %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}

indexer\_config  
grok {  
  break_on_match => false  
  match => [ "message","%{NGINXACCESSA}" ]  
  match => [ "message","%{NGINXACCESSB}" ]  
}

[bravo-kernel]

Bravo Kernel [DISQUS]: You might be right but I can assure you that I have tested the identically named patterns in real life without grok failures. A kind request.. your code examples are unreadable here. Could you update your comment and replace them with a gist-link? TIA

[bravo-kernel]

mrwpinks [DISQUS]: Hey, your examples are now failing on Grok Debugger. I had this working fine previously but now I'm getting no matches, has something changed?

[bravo-kernel]

Bravo Kernel [DISQUS]: Not that I'm aware of but I haven't updated anything for a while so I really can't tell. Please share if you find the cause.

[bravo-kernel]

mrwpinks [DISQUS]: shall look at this deeper tomorrow morning (fun times ahead)

[bravo-kernel]

czerasz [DISQUS]: I really recommend to also read this article: http://logz.io/blog/nginx-log-analysis/