Add log type include filter by jorbaum · Pull Request #714 · cloudfoundry/loggregator-agent-release

jorbaum · 2026-01-05T14:06:49Z

Description

Addresses #711 :

Support both ?include-log-source-types= and ?exclude-log-source-types (calling it log source type instead of log type as log type is too generic and is already used in this code base)
Throw error in case both are configured. Do not throw an error on misconfigured values for the new config option (consistent with previous HTTP query param usage in binding config), but log a warning (new).
Only exclude filter shall pass unknown/nil source type, include filter will discard

String parsing should be efficient enough dues to:

running only when source type tag and log type filter is present
strings.IndexByte might take some time, but it is used only with a single char / and strings.IndexByte() should be more efficient than strings.Index()
Looking up content in a map logTypePrefixes rather than in each log line, which should be more efficient

Related PRs:

Disclaimer: Claude helped me in crafting this code.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing performed?

Unit tests
Integration tests
Acceptance tests

Checklist:

This PR is being made against the main branch, or relevant version branch
I have made corresponding changes to the documentation
I have added testing for my changes

chombium

@jorbaum Thanks for the changes.

Generally it looks good, but I have some smaller objections which should be addressed.

chombium · 2026-01-09T17:05:15Z

@ZPascal Wdyt about this PR? It addresses #711

jorbaum · 2026-01-12T08:04:26Z

@chombium I left a bunch of replies for points that are not yet clear to me. All the rest look good.

Thanks for the thorough review! You found a bunch of nice things.

jorbaum

Worked in some comments.

jorbaum

Worked in most comments. Still 2 left to clarify.

jorbaum · 2026-01-19T15:30:27Z

Feel free to take another look at it.

chombium

Hi @jorbaum. Thanks for the improvements. Unfortunately, I've found few more problematic things.

jorbaum · 2026-02-18T14:45:29Z

Thanks for the review. I replied to a bunch of comments for some clarifying remarks. Also a quick discussion about uppercase/lowercase config values would help me.

jorbaum

Addressed hopefully all comments.

I got a bunch of things to clarify. Biggest question that is left is likely if we should recommend users uppercase or lowercase values.

chombium

@jorbaum It looks much better now and we're close to mergnig.
Please fix the small things which I've commented on.

I will have to test everything before merging.

chombium · 2026-03-16T14:01:40Z

+			}
+		})
+
+		It("filters out the log with missing source_type when include filter is configured", func() {


The filter works based on the values of the source_type tag and here the code defines the behavior of what should happen.

Is this behavior documented somewhere?

No it is not. The best place for that is likely https://docs.cloudfoundry.org/devguide/services/log-management.html ?

Yes, that's the place. Please open a PR. Btw, why is the description of the parameters already in the docs?

Yes, that's the place. Please open a PR.

Opened up a new PR at cloudfoundry/docs-dev-guide#584

Btw, why is the description of the parameters already in the docs?

The previous PR has been merged a bit sooner than expected.

jorbaum

Should all be addressed now. CI is failing due to new golangci-lint changes. Those have been addressed in the two newest commits of #717 .

jorbaum · 2026-03-17T16:37:18Z

+			}
+		})
+
+		It("filters out the log with missing source_type when include filter is configured", func() {


Yes, that's the place. Please open a PR.

Opened up a new PR at cloudfoundry/docs-dev-guide#584

Btw, why is the description of the parameters already in the docs?

The previous PR has been merged a bit sooner than expected.

jorbaum · 2026-03-27T08:57:12Z

I created a new branch that I rebased on #633 just in case we need it. Should also be ready for review.

maecval · 2026-06-19T05:10:14Z

@jorbaum @chombium
Hi all,
We would love to have this in!
We already relied on the documentation and implemented a change on this. Which of course did not work as expected (got too many APP logs...).
Hope to see this merged soon!

chombium · 2026-06-19T12:19:52Z

Hey @maecval,

This change will be merged most probably next week as it needs a bit more testing. Sorry for keeping you waiting :-/

jorbaum · 2026-06-19T12:24:14Z

@maecval sorry to hear that. There is also a documentation draft PR that reflects the most up to date changes of this branch.

maecval · 2026-06-22T05:51:44Z

@jorbaum & @chombium
Thank you guys for pushing this forward!
No need to be sorry - we all know how things like that can happen.

I noticed that as my changes depend on `RawQuery` not being empty, but actually they were.

Also rename remaining instances of log type to source type

jorbaum · 2026-07-10T15:44:59Z

Rebased on the newest changes also updated PR description.

chombium · 2026-07-16T15:44:51Z

Hi @jorbaum,

I did few tests with the new filter and I have that when the include-log-source-types and exclude-log-source-types=APP are used alone, without any other parameters, everything works as expected.
Examples syslog drain URLs:

https://test-drain.cf.obi-wan.env.bndl.sapcloud.io?include-log-source-types=RTR&exclude-log-source-types=APP
https://test-drain.cf.obi-wan.env.bndl.sapcloud.io?exclude-log-source-types=APP
https://test-drain.cf.obi-wan.env.bndl.sapcloud.io?include-log-source-types=APP

Then I did few tests in which I wanted to see if I get everything: container metrics and app logs except the APP logs

https://test-drain.cf.obi-wan.env.bndl.sapcloud.io?drain-data=all&exclude-log-source-types=APP
https://test-drain.cf.obi-wan.env.bndl.sapcloud.io?drain-data=all&include-log-source-types=APP

I did get everything, including the APP logs.

I guess if a value of drain-data is set, the include and exclude parameters have no effect.

Please check how the parameters work along with the other Syslog Drain parameters.

jorbaum · 2026-07-17T12:00:48Z

@chombium

You are correct. Filtering was only configurable when drain-data was not set to all. I changed it now that:

filters will be applied for drain-data being logs or all and for drain-type being logs or all
filters will now also be applied for loggregator envelope events. Note that events will only be forwarded when drain-data parameter is used. I did not change that logic.
Instead, I clarified this in the code a bit.

I also added and successfully ran a new acceptance test for drain-data=all

chombium

Hi @jorbaum,

Everything looks good a part from one thing:
Please exchange include-log-source-types and exclude-log-source-types with include-log-types and exclude-log-types as we don't have types of log sources, but log types. This way we will keep the current narative as in the CF Docs Log types and their messages page.

cf-foundation-community-automation Bot added this to Application Runtime Platform Working Group Jan 5, 2026

cf-foundation-community-automation Bot moved this to Inbox in Application Runtime Platform Working Group Jan 5, 2026

jorbaum force-pushed the add-log-type-filter branch 2 times, most recently from 9fcc372 to bccd92c Compare January 8, 2026 09:43

chombium requested changes Jan 9, 2026

View reviewed changes

jorbaum commented Jan 12, 2026

View reviewed changes

Comment thread src/pkg/egress/syslog/filtering_drain_writer.go Outdated

Comment thread src/pkg/ingress/bindings/binding_config.go Outdated

Comment thread src/pkg/ingress/bindings/binding_config.go Outdated

jorbaum commented Jan 13, 2026

View reviewed changes

Comment thread src/pkg/ingress/bindings/binding_config.go Outdated

Comment thread src/pkg/ingress/bindings/binding_config_test.go Outdated

jorbaum commented Feb 4, 2026

View reviewed changes

Comment thread src/pkg/egress/syslog/log.go Outdated

jorbaum force-pushed the add-log-type-filter branch 3 times, most recently from 108740d to ef16bed Compare February 11, 2026 14:33

jorbaum marked this pull request as ready for review February 16, 2026 10:18

jorbaum requested a review from a team as a code owner February 16, 2026 10:18

This was referenced Feb 16, 2026

Document new log-source-type syslog drain config cloudfoundry/docs-dev-guide#576

Merged

Document more source types cloudfoundry/docs-dev-guide#577

Draft

chombium requested changes Feb 17, 2026

View reviewed changes

jorbaum commented Feb 20, 2026

View reviewed changes

jorbaum force-pushed the add-log-type-filter branch from ef16bed to ecab32a Compare February 20, 2026 13:38

chombium requested changes Mar 16, 2026

View reviewed changes

jorbaum mentioned this pull request Mar 17, 2026

Improve log type filter cloudfoundry/docs-dev-guide#584

Draft

jorbaum commented Mar 17, 2026

View reviewed changes

jorbaum added 20 commits July 10, 2026 14:09

Move next test to table

9b7d458

Move drain data for old parameter test to table

ba2b7ab

Move log filter validation to binding_fetcher

f5aa7bc

Dereference pointer to make an actual copy

da09b15

I noticed that as my changes depend on `RawQuery` not being empty, but actually they were.

Rename log type to source type where possible

9d442da

Only exclude filter shall pass unknown source type

e733792

Also rename remaining instances of log type to source type

Add SourceType PROXY, HEALTH, SYS and STATS

a1ec8d1

Make missing source_type tests clearer

dd42633

Add tests for no filtering value configured

9f7a5df

Rename source to log source where sensible

a520a28

Remove unused method

7ba8c60

Add -log to uri query param

4184c3c

No longer accept URI drains with space in values

f1bb2a0

Move log source type parsing into common function

6385a74

Group tests where source_type is present

ee5b2b8

Use new package name in test

efb81ed

Add filtering validation to poller

6ff68b3

Fix cyclomatic complexity by refactoring poller

97231b5

Also move filtering tests to poller

c90ecf3

Follow refactoring of FilterebBindingFetcher

86b5879

jorbaum force-pushed the add-log-type-filter branch from a312cd4 to 86b5879 Compare July 10, 2026 15:39

jorbaum mentioned this pull request Jul 15, 2026

Add syslog filter test cloudfoundry/cf-acceptance-tests#2033

Draft

7 tasks

jorbaum added 3 commits July 17, 2026 10:42

Support all

af1453a

Filter envelopes by source type just like logs

2143322

Clarify event forwarding via drain-type param

a014672

Add filter test for multiple source type case

ad5f2a6

chombium requested changes Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

jorbaum commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Testing performed?

Checklist:

Uh oh!

chombium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chombium commented Jan 9, 2026

Uh oh!

jorbaum commented Jan 12, 2026

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jorbaum commented Jan 19, 2026

Uh oh!

Uh oh!

chombium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorbaum commented Feb 18, 2026

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chombium left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chombium Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

jorbaum Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

chombium Mar 16, 2026

Choose a reason for hiding this comment

jorbaum commented Jan 5, 2026 •

edited

Loading

chombium left a comment •

edited

Loading

chombium commented Jun 19, 2026 •

edited

Loading

jorbaum commented Jun 19, 2026 •

edited

Loading

maecval commented Jun 22, 2026 •

edited

Loading

jorbaum commented Jul 10, 2026 •

edited

Loading

jorbaum commented Jul 17, 2026 •

edited

Loading