TL;DR: See title. How can I tell Google they’re probably processing their mail wrong?

After setting up the Matrix Authentication Service (MAS) and exim-relay as mail server, I noticed verification mails sent from the service are often in the spam directory.

When digging deeper, I found out the mails are failing DKIM authentication. This was weird because DKIM is set up correctly, as verified by other mail providers and online DKIM test tools such as DMARC Tester.

Searching online for “gmail fails DKIM authentication, while other providers pass”, I found regular reports, posts or similar without resolution, or unrelated resolutions such as DKIM alignment.

Using meld, I compared the original source of mails as received by gmail with those of other providers, and found a difference:

In other providers, the header for “From:” and “Reply-To:” fields are presented with double-quotes:

From: "John Smith" <j.smith@example.com>
Reply-To: "John Smith" <j.smith@example.com>

In gmail, where DKIM fails, there are no double-quotes:

From: John Smith <j.smith@example.com>
Reply-To: John Smith <j.smith@example.com>

As this should be the raw source each, I ruled out presentation issues and digged deeper.

I found out, that specifically the rust crate lettre, as used by the MAS, encodes names with whitespace using double-quotes. Further, from researching a bit more and reading RFC 2822 sections 3.2.4 and 3.2.5, I come to the conclusion that whitespace needs no quoting in mail headers.

I created issues upstream and downstream to report the issue at lettre and MAS, particularly that their mails are failing DKIM checks at gmail:

If you’ve read that far, you probably wonder why I post all of that? For one, to provide another data point for people scratching their heads over mail issues.

But other than that: I’m pretty sure the google mail servers should not strip the quotes before doing the DKIM check. I assume they have some kind of decode -> process -> encode workflow, that then simply encodes the headers again, this time without the quotes. But IMHO a correctly signed message should not lead to an authentication error, even if the contents are not perfectly encoded.

I would be curious on getting some feedback from some mail experts on what is happening here. This is not my field of expertise and I’m going by what I’ve learned over the past 48h.

  • voracitude@lemmy.world
    3·
    4 hours ago

    Yeah, sounds like you cracked it, frankly. This does make sense - the signature covers the entire message, including the original headers (which would include those quotes, in your case). If Google’s processing is removing quotes that were originally there, the message changed, so DKIM would fail.

    I think Google must only leave the quotes in if there are special characters like a comma in the name. Are you able to update your exim to only include quotes when the from name includes a special character? Feels bad having to engineer around Google’s incompetence, I know, but that should solve the issue.

    Edit: Or you could change your “from” name to “no @ reply” or “daemon @ box” or something, which would force Google to leave the quotes in.

    • w2xel@gehirneimer.deOP
      2·
      6 hours ago

      I think it’s technically an encoding bug in lettre, which is used in matrix authentication services: https://github.com/lettre/lettre . As far as I can tell, exim is relaying the messages “correctly” or at least without altering them.

      I.e. lettre should not add quotes for whitespace. But also google shouldn’t alter messages before authenticating. In an ideal world, both sides are fixed ^^

      • hperrin@lemmy.caEnglish
        2·
        3 hours ago

        Lettre is following the spec, where if the display name contains a space, the modern way to encode it is to put quotes around it.

      • voracitude@lemmy.world
        2·
        6 hours ago

        Sorry, was editing while you were replying - I’ll reply here in case you don’t see my edits, sorry if you read them already. In light of the following:

        • The only difference in the emails is the quotes in the “From”
        • DKIM passes after you manually add the quotes back in
        • Every other provider leaves the quotes in place

        I’m pretty sure that your setup is fine, including your exim config, and this is an issue specifically with Google’s processing like you originally thought. Try making your “from” name for the service “daemon @ box”, that will force Google to leave the quotes in. If the email passes dkim at Gmail like that, we have definitively proven your original theory correct.