When full_message is <250 chars, short_message vanishes in an undocumented way

[mrmanc]

One of our engineers was frustrated that they could not develop an aggregated Kibana visualisation as they had hoped to. They wanted to look at the frequency of errors grouped by their message. They could not use the message field as it shows as non-aggregatable in Elastic (presumably due to its length). They could not use the short_message field since it is empty when the message field is not truncated.

It took me some time to track this back to this conditional in the plugin code, as I initially assumed it was the Logstash GELF library that is used by the client that was leaving the short_message out. The docs for this plugin do not indicate that the remap functionality will actually remove the short_message in the event it matches the message field.

It seems to me that the short_message should be preserved, or that behaviour should be properly documented.

• Version: 6.2.2
• Operating System: CentOS
• Config File (if you have sensitive info, please remove it):

input {
  gelf {
    id => "application_logs_gelf"
    port => 4207
    remap => true
  }
}
output {
  stdout { codec => rubydebug }
}

• Sample Data:

{
  "version": "1.1",
  "host": "example.org",
  "short_message": "This is less than 250 characters.",
  "full_message": "This is less than 250 characters.",
  "timestamp": 1385053862.3072,
  "level": 1,
  "_user_id": 9001,
  "_some_info": "foo",
  "_some_env_var": "bar"
}

• Steps to Reproduce:

Feed the sample data to the gelf input plugin and observe that the output does not contain short_message.