Skip to content

A non-ASCII comment loses its parentheses when a header is folded with an encoded word #162125

Open

Description

@FedeLoch

Bug report

Hello,

While fuzzing the email package of Python 3.14.7 with Termo, a fuzzer I'm developing, I checked that a header value survives as_string() and parsing back with policy.default. Structured headers that contain a comment with non-ASCII text don't survive it. The comment is written as an RFC 2047 encoded word without its parentheses, so on reading back it is no longer a comment. In an address header this changes the addresses.

import email
from email import policy
from email.message import EmailMessage

msg = EmailMessage()
msg["From"] = "j@example.com (José)"
out = msg.as_string()
print(out.splitlines()[0])
# From: j@example.com =?utf-8?q?Jos=C3=A9?=
print(str(email.message_from_string(out, policy=policy.default)["From"]))
# j@example.com, <>        <- one sender became two addresses, the second empty

msg = EmailMessage()
msg["To"] = "Ana (ñ) <a@example.com>"
out = msg.as_string()
print(out.splitlines()[0])
# To: Ana =?utf-8?q?=C3=B1?= <a@example.com>
print(email.message_from_string(out, policy=policy.default)["To"].addresses[0].display_name)
# Ana ñ                    <- the comment is now part of the display name

msg = EmailMessage()
msg["Content-Disposition"] = "attachment (é)"
print(msg.as_string().splitlines()[0])
# Content-Disposition: attachment =?utf-8?q?=C3=A9?=

With an ASCII comment the parentheses are kept (From: j@example.com (Jose)), and RFC 2047 §5 (2) allows an encoded word inside a comment. So the expected output is, for example, To: Ana (=?utf-8?q?=C3=B1?=) <a@example.com>.

Versions:

  • 3.14.7: all three as above.
  • 3.15.2: the To and Content-Disposition cases are the same. The From case now raises HeaderWriteError: Non-ASCII domain 'example.com (José)' is invalid under current policy setting (utf8=False), and "Jürgen <j@example.com> (café)" raises Non-ASCII cfws ' (café)' is invalid. The comment is not part of the domain, so this error is also misleading.
  • main (5a22a62): I haven't run it, but the code path below is unchanged.

The cause appears to be in _refold_with_ew (Lib/email/_header_value_parser.py). When a part needs encoding because it has non-ASCII characters, a non-terminal part is unpacked into its children (line 2973, "It's not a Terminal, do each piece individually"). For a Comment, the children don't include the ( and ), which only Comment.__str__ adds. The parenthesis handling added for gh-143935 (line 3036, if part.token_type == 'comment':) is only on the path for parts too long to fit on a line, not on this one. Wrapping the children the same way there (an opening ( terminal, the children, then a closing ), with the encoded words kept inside) should fix it.

I can turn these into test cases for test__header_value_parser / test_headerregistry if that helps.

If you'd like me to keep fuzzing the email package (other headers and policies, or re-checking after a fix), feel free to reach out. I'd be glad to keep working on it with you.

Thanks!

PS: Sorry if the issue looks like a template, it's, I am reporting plenty of issues to different projects and I am a simple human :)

CPython versions tested on:

3.14

Operating systems tested on:

macOS

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions