diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md new file mode 100644 index 00000000..45c9e22d --- /dev/null +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -0,0 +1,211 @@ +# Fluentd v1.17.0 has been released + +Hi users! + +We have released v1.17.0 on 2024-04-30. ChangeLog is [here](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430). + +This release is a new release of v1.17 series. +In this release, we added some new features for some plugins and fixed bugs of Parser. + +## Enhancement + +### `in_tail`: Add `glob_policy` option for expanding glob capability of `path` and `exclude_path` + +In this release, we added a new option [glob_policy](https://docs.fluentd.org/input/tail#glob_policy) for [in_tail](https://docs.fluentd.org/input/tail) plugin. + +In previous versions, we can use only `*` in glob patterns for [path](https://docs.fluentd.org/input/tail#path) and [exclude_path](https://docs.fluentd.org/input/tail#exclude_path) option. + +Example: + +``` +path /path/to/* +exclude_path ["/path/to/*.gz", "/path/to/*.zip"] +``` + +From this version, we can also use `[]`, `?`, and `{}` in glob patterns depending on the `glob_policy` option. + +Example: + +``` +path "[0-1][2-3].log" +glob_policy extended +``` + +Please see [the document](https://docs.fluentd.org/input/tail#glob_policy) and [#4401](https://github.com/fluent/fluentd/pull/4401) for more information. + +### `out_http`: Support AWS Signature Version 4 authentication + +In this release, we added a new option `aws_sigv4` for the [method](https://docs.fluentd.org/output/http#method) setting of [out_http](https://docs.fluentd.org/output/http) plugin. + +By using this option, `out_http` can use [AWS Signature Version 4](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). + +For example, this allows `out_http` to write to [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html). + +Please see [the document](https://docs.fluentd.org/output/http#method) and [#4459](https://github.com/fluent/fluentd/pull/4459) for more information. + +### `out_http`: Add option to reuse connections + +In this release, we add a new option [reuse_connections](https://docs.fluentd.org/output/http#reuse_connections) for [out_http](https://docs.fluentd.org/output/http) plugin. + +This option will improve throughput of `out_http`. + +Please see [the document](https://docs.fluentd.org/output/http#reuse_connections) and [#4330](https://github.com/fluent/fluentd/pull/4330) for more information. + +### `in_http`: Recognize CSP reports as JSON data + +In this release, we make the data type of the request where the `Content-Type` is `application/csp-report` be considered JSON by default. + +Now, `in_http` can receive [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)'s report by default. + +Please see [#4282](https://github.com/fluent/fluentd/pull/4282) for more information. + +## Bug Fixes + +### Make sure parser returns hash + +The record data in an event of Fluentd must be a hash object. + +* [Life of a Fluentd event](https://docs.fluentd.org/quickstart/life-of-a-fluentd-event) + +However, in the previous versions, some parser plugins could return a non-hash object, such as an array. +It could cause errors in subsequent processing. + +In this release, the following parser plugins have been fixed. + +* [parser_json](https://docs.fluentd.org/parser/json) +* [parser_msgpack](https://docs.fluentd.org/parser/msgpack) + +The changes are as follows: + +* Make sure to return a hash record. +* Make sure to accept only a hash or an array of hash. + +Here are the details for each case. + +#### Example of changed behavior + +Config: + +``` + + @type tcp + tag test.tcp + + @type json + + + + + @type stdout + +``` + +Send an array data: + +``` +$ netcat 0.0.0.0 5170 +[{"k":"v"}, {"k2":"v2"}] +``` + +The result before this version: + +``` +{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}] +``` + +The result after this version: + +``` +{datetime} test.tcp: {"k":"v"} +{datetime} test.tcp: {"k2":"v2"} +``` + +#### Example of resolved error + +Config: + +``` + + @type tcp + tag test.tcp + + @type json + null_empty_string + + + + + @type stdout + +``` + +Send an array data: + +``` +$ netcat 0.0.0.0 5170 +[{"k":"v"}, {"k2":"v2"}] +``` + +The result before this version: + +``` +{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array" +``` + +The result after this version: + +``` +{datetime} test.tcp: {"k":"v"} +{datetime} test.tcp: {"k2":"v2"} +``` + +#### Remaining problem: filter_parser + +In the previous versions, `filter_parser` could return an array record based on this wrong behavior. +From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. +This behavior should improve in the future. + +Here is an example. + +``` + + @type sample + tag test.array + sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"} + + + + @type parser + key_name message + + @type json + + + + + @type stdout + +``` + +The result before this version: + +``` +{datetime} test.array: [{"k":"v"},{"k2":"v2"}] +``` + +The result after this version: + +``` +{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"} +{datetime} test.array: {"k":"v"} +``` + +These are the major changes for this release. + +In addition, some performance improvements have been included! +Please see [ChangeLog](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430) for details! + +Enjoy logging! + +TAG: Fluentd Announcement +AUTHOR: clearcode