kv split data within value

[Mister-X-]

kv parses just fine values within double quotes (and probably also with single quote). However, it parses the value as key-value if the value contains equal signs. This can be reproduced with an empty kv filter such as:

filter { kv { } }

With the following item, the value in cgfattr get split further and we end up with weird keys in Elastic Search:

date=2015-05-18 time=14:13:48 devname=FG140XXXX devid=FG140XXXXT logid=0100044547 type=event subtype=system level=information vd="VirtualDomain" logdesc="Configure object attribute" user="user1" ui="jsconsole" action=Edit cfgtid=1790967809 cfgpath="system.wccp" cfgobj="101" cfgattr="password[ENC K7taRRarXYdpNvARTqktIeNcecPbJB6gsRQPLKjjftFAj81qnhoGStE4PKI9PGjYodn/Z/f26bcGG0FDpsq4scGzMONwrNuV973xkizVF/YawO8kDmdAlCJeFVHJG99J1gwVhxqjz6cmWSF5aI6FcgAfyk4gjh4yJe0p/oWks3bXxCT2Q/6juahXAIqBtIY9ZJCMJw==->ENC K7taRTt0SmYF1SbAdZes1UJbKzwzFyD/0nrlQ6JHH0Dir6kdtDCtdrT5f9/GfwxmkmAS7hNS+Tidmrrczxf1FNdedgIQIt6gVx+C1J63RWtOp+D68aDScOgBXkO05An3o8EGo4+GyYIr1yUtG1QEGYlbJ7ZhTkYNmpxCi55PdHFZeGROMjvhfB7cSwsGjHpskFk4ug==]" msg="Edit system.wccp 101"

[jordansissel]

I can't reproduce this in a test case:

The only fields I see are these:

timestamp, @version, action, cfgattr, cfgobj, cfgpath, cfgtid, date, devid, devname, level, logdesc, logid, message, msg, subtype, time, type, ui, user, vd

[jordansissel]

I can't reproduce this with Logstash 1.5.0, either: https://gist.github.com/jordansissel/01d8686b09ef31692fc6

[mpcokeefe]

I am seeing the same problem with Logstash 2.3.1. I have firewall logs containing key value pairs, where each value is contained in double quotes. If a value contains an = sign as in a URL Logstash splits the value into a key value pair.

[mpcokeefe]

Because the equal sign in the KV pair always comes before a quote, I can differentiate them from the equals sign within the value, so a workaround is to replace them with a different symbol to use as the value_split.

This works as long as I never get a URL which contains either an equals-quote anywhere, or the replacement symbol I used below:

mutate {
    gsub => [ "message", '="', '‡"' ]
}
kv {
    value_split => "‡"
}