ares_parse() fails on outlook.com/alum.mit.edu added header

[mdomsch]

opendkim 2.11.0-Beta2 reports in maillog:
failed to parse Authentication-Results: header field

I patched it to include the offending header content in the syslog to find it:

Authentication-Results: spf=pass (sender IP is 209.85.219.172)#12 smtp.mailfrom=example.org; alum.mit.edu; dkim=pass (signature was verified)#12 header.d=example.org;alum.mit.edu; dmarc=pass action=none#012 header.from=example.org;compauth=pass reason=100

This header is being added by MX alum-mit-edu.mail.protection.outlook.com which receives mail on behalf of the MIT alum.mit.edu email forwarding service, before forwarding on to the true destination address. At that true destination, opendkim reports its inability to parse this header.

When mail is sent directly from example.org to the true destination address (not going through the alum.mit.edu forwarder), the above header is not added, and opendkim works as expected.

[mdomsch]

The mail is being received for alum.mit.edu by Office365, which adds the malformed AR header, and keeps it there when forwarded to its final destination.

[martinbogo]

@mskucherawy and I will review.

[mdomsch]

It fails because outlook.com/Office 365 is not including it's own host name as the first item in the header, as it should. I don't know why that is, but when I see the message delivered also into the outlook.com mailbox, it has the added AR header without the host name as the first item.

[mdomsch]

jstedfast/MimeKit#490 notes the same problem.

[mdomsch]

https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/email-validation-and-authentication?view=o365-worldwide describes their creation of the header. No authserv-id is ever shown.

[mikenor]

This looks like a duplicate of #48.

[kenfc]

OpenDKIM header read failures with emails from Microsoft services has been a problem for me for a very long time,

I keep hoping to see a project update or at least a patch for the issue. But to be honest, I'm beginning to wonder if the project has been abandoned

[maxfieb]

This happens with all o365 signed emails. They omit the ADMD on DKIM signatures. Arbitrary example :

Authentication-Results:
compauth=<fail | pass | softpass | none> reason=

OpenDKIM only understands AR headers with the ADMD; in place. For DKIM the ADMR seems optional out in the world.

Authentication-Results: thefunkychecken.com;
compauth=<fail | pass | softpass | none> reason=

Its a bit hard to follow, but it seems like the parser in opendkim-ar.c assumes there will always be an ADMD present

Google etc., accept the Authentication-Results from o365 without an error, as do o365 instances.

For DKIM at least, seems like the starting "something;" token(s) after the header name, should be optional.

[thegushi]

Thanks for the detailed reports. To clarify what's happening here: OpenDKIM is actually behaving correctly.

RFC 8601 requires the Authentication-Results header to begin with an authserv-id (the hostname of the system that performed the checks), followed by a semicolon, before any method results. The headers generated by Office 365 omit this field entirely, jumping straight to spf=pass ... - which is not valid per the spec.

When OpenDKIM's parser encounters this, it logs a warning and skips the header. No mail is rejected or deferred as a result; the only impact is the log noise.

We understand this is annoying, and we're sympathetic - O365 sends a lot of mail. If there's a strong case for adding a lenient parsing mode that gracefully handles authserv-id-less headers, we're open to it, but we'd want to see whether that creates any unintended side effects before going down that road.

In the meantime: this is a Microsoft standards compliance issue, not an OpenDKIM bug. You may want to filter the warning from your logs if the volume is a problem.

[maxfieb]

Microsoft does not follow the standards opendkim uses, wether intentional or not it omits the ADMD (first parameter source.host.com;) on Authentication-Results: that microsoft provides that is invalid by spec (when microsoft originates mail and you are trying to receive it), so open(dkim|arc) which share code from ares_parse() and parses to spec, can cascades to a validation failure reported by opendmarc (which summarises the open(dkim|arc) results.

It's likely Microsoft is leaks it's own internal Authentication-Results header (for inbound mail) via outbound mail, the Authenication-Results header misrosoft uses internally is not RFC compliant, and internally the ADMD(s) would only knowable after the mail is mapped to a Microsoft tenancy, so it needs to be omitted internally. They have a deep cluster structure, so imho it makes sense it's an internal header that "escaped" or repeated outbound somehow.

Because OpenDKIM strictly adheres to RFC-compliant ABNF state tracking, ares_parse() aborts execution immediately when it finds an invalid method tag, or where it expected an ADMD string. This means if both opendkim and openarc abort parsing the ABNF syntax for the Authentication-Results without ADMD and opendmarc, will (correctly?) summarise that as an authentication fail, as it summarises the success of the previous checks.

Opendkim as the first step in the opendkim/openarc/opendkim chain (when validating only) could ignore a provided inbound Authentication-Results from a security perspective (as validation by the sending host should be untrusted), also because when OpenDKIM encounters the malformed Microsoft header, ares_parse() has a hard error in opendkim and openarc that does skips validation? In this case, which opendmarc then dutifully reports as a failure (especially if DMARC policy=reject).

One way might be to to test for non-conforming Authentication-Results using the existing ABNF rules, in the way of ares_parse() but ares_check() same arguments but returns true if it is a opendkim parseable header, opendkim could then have an option to detect/remove invalid incoming Authentication-Results in validation mode.

( in the validation case, a new Authentication-Results header will be created anyway by opendmarc, if opendkim/openarc are protected from the bogus Microsoft value, and both do their part ).

[thegushi]

Also, see my recent post on mailop where O365 is rejecting valid forwarded mail because DMARC passes, but SPF fails. They've been breaking standards for years and I dislike that they have just become "big mailbox" that everyone has to tiptoe around. That's doesn't make me thrilled to be working an issue they caused. But....