TL;DR: See title. How can I tell Google they’re probably processing their mail wrong?
After setting up the Matrix Authentication Service (MAS) and exim-relay as mail server, I noticed verification mails sent from the service are often in the spam directory.
When digging deeper, I found out the mails are failing DKIM authentication. This was weird because DKIM is set up correctly, as verified by other mail providers and online DKIM test tools such as DMARC Tester.
Searching online for “gmail fails DKIM authentication, while other providers pass”, I found regular reports, posts or similar without resolution, or unrelated resolutions such as DKIM alignment.
Using meld, I compared the original source of mails as received by gmail with those of other providers, and found a difference:
In other providers, the header for “From:” and “Reply-To:” fields are presented with double-quotes:
From: "John Smith" <j.smith@example.com>
Reply-To: "John Smith" <j.smith@example.com>
In gmail, where DKIM fails, there are no double-quotes:
From: John Smith <j.smith@example.com>
Reply-To: John Smith <j.smith@example.com>
As this should be the raw source each, I ruled out presentation issues and digged deeper.
I found out, that specifically the rust crate lettre, as used by the MAS, encodes names with whitespace using double-quotes. Further, from researching a bit more and reading RFC 2822 sections 3.2.4 and 3.2.5, I come to the conclusion that whitespace needs no quoting in mail headers.
I created issues upstream and downstream to report the issue at lettre and MAS, particularly that their mails are failing DKIM checks at gmail:
- https://github.com/lettre/lettre/issues/1125
- https://github.com/element-hq/matrix-authentication-service/issues/5497
If you’ve read that far, you probably wonder why I post all of that? For one, to provide another data point for people scratching their heads over mail issues.
But other than that: I’m pretty sure the google mail servers should not strip the quotes before doing the DKIM check. I assume they have some kind of decode -> process -> encode workflow, that then simply encodes the headers again, this time without the quotes. But IMHO a correctly signed message should not lead to an authentication error, even if the contents are not perfectly encoded.
I would be curious on getting some feedback from some mail experts on what is happening here. This is not my field of expertise and I’m going by what I’ve learned over the past 48h.
Quotes around the display name in a header certainly conforms to the standard (and in fact, the way Gmail rewrites it does not conform to the standard when it contains a space, but is the obsolete form), and I would expect any decent mail program to leave it alone. Then again, Gmail is not a decent mail program, and hasn’t been for a long time.
Edit: @[email protected] points out below that Gmail’s rewrite does follow the spec, so it is conforming to the standard when white space is in the display name. It’s only when there is a dot (.) that it would be using the obsolete form.
I had to think about this for a while, but the standard is only obsoleting folding white space, i.e. white space that wraps lines, such as:
Subject: This is a folded linewhich is equivalent to
Subject: This is a folded lineAs I understand it, white space is allowed before and after obsoletion. Or do I understand it wrong?
Edit:
I think in the obsoleted language the following would have been allowed for a From: field as well:
From: John Smith <j.smith@example.com>Folding white space is a different issue. It’s the
atom / quoted-stringpart. That’s the standard form of a display name. Meaning if it’s more than one word (separated by white space), it should use the quoted string form. It does listobs-phraseas an alternative, but using obsolete syntax should be avoided when possible.phraseis1*word, notword. The difference toobs-phraseis that it allows dots (“.”) and folding whitespace. Not that it allows whitespace.Atom is defined as such:
atext = ALPHA / DIGIT / ; Any character except controls, "!" / "#" / ; SP, and specials. "$" / "%" / ; Used for atoms "&" / "'" / "*" / "+" / "-" / "/" / "=" / "?" / "^" / "_" / "`" / "{" / "|" / "}" / "~" atom = [CFWS] 1*atext [CFWS]So since atom does not contain white space, there should be no white space. Only comments can separate
1*atom.If they wanted to allow separating atoms by a space, it would be written:
atom *(SP atom)Edit: wait, I’m wrong. CFWS includes white space. You’re right.
Ah, but I was right for the wrong reasons, I also only now understand it fully. Thanks for the nice discussion.
The unquoted names with whitespace are legal, as stated in A.5: https://www.rfc-editor.org/rfc/rfc2822#appendix-A.5
The section of obsolete syntax in the appendix starts just below A.5., unfortunately.
I agree though, that a quoted string seems to be perfectly legal as well, so there is no good reason for google to change it.
The section right below shows that it’s obsolete:
A.6.1. Obsolete addressing Note in the below example the lack of quotes around Joe Q. Public, the route that appears in the address for Mary Smith, the two commas that appear in the "To:" field, and the spaces that appear around the "." in the jdoe address. ---- From: Joe Q. Public <john.q.public@example.com> To: Mary Smith <@machine.tld:mary@example.net>, , jdoe@test . example Date: Tue, 1 Jul 2003 10:52:37 +0200 Message-ID: <5678.21-Nov-1997@example.com> Hi everyone. ----And in the spec itself, that syntax is named as “obs-phrase”.
But yes, though obsolete, it is still legal syntax. So I guess I shouldn’t say it “does not conform to the standard”, but rather “just barely conforms to the standard”.
In the same example, the
Mary Smithis not the issue, rather the@machine.tld, as is written in the description.John Q. Publicis an issue because of the dot, not the spaces. The spaces are an issue injdoe@test . example, as that’s actually an address, not a nameYep, you are right. So Gmail is following the spec, but still altering headers. So at least it’s not breaking the email, just breaking the authentication.