Repository navigation
Description
Bug report
Hello,
While fuzzing the email package of Python 3.14.7 with Termo, a fuzzer I'm developing, I checked that a header value survives as_string() and parsing back with policy.default. Structured headers that contain a comment with non-ASCII text don't survive it. The comment is written as an RFC 2047 encoded word without its parentheses, so on reading back it is no longer a comment. In an address header this changes the addresses.
import email
from email import policy
from email.message import EmailMessage
msg = EmailMessage()
msg["From"] = "j@example.com (José)"
out = msg.as_string()
print(out.splitlines()[0])
# From: j@example.com =?utf-8?q?Jos=C3=A9?=
print(str(email.message_from_string(out, policy=policy.default)["From"]))
# j@example.com, <> <- one sender became two addresses, the second empty
msg = EmailMessage()
msg["To"] = "Ana (ñ) <a@example.com>"
out = msg.as_string()
print(out.splitlines()[0])
# To: Ana =?utf-8?q?=C3=B1?= <a@example.com>
print(email.message_from_string(out, policy=policy.default)["To"].addresses[0].display_name)
# Ana ñ <- the comment is now part of the display name
msg = EmailMessage()
msg["Content-Disposition"] = "attachment (é)"
print(msg.as_string().splitlines()[0])
# Content-Disposition: attachment =?utf-8?q?=C3=A9?=With an ASCII comment the parentheses are kept (From: j@example.com (Jose)), and RFC 2047 §5 (2) allows an encoded word inside a comment. So the expected output is, for example, To: Ana (=?utf-8?q?=C3=B1?=) <a@example.com>.
Versions:
- 3.14.7: all three as above.
- 3.15.2: the
ToandContent-Dispositioncases are the same. TheFromcase now raisesHeaderWriteError: Non-ASCII domain 'example.com (José)' is invalid under current policy setting (utf8=False), and"Jürgen <j@example.com> (café)"raisesNon-ASCII cfws ' (café)' is invalid. The comment is not part of the domain, so this error is also misleading. - main (5a22a62): I haven't run it, but the code path below is unchanged.
The cause appears to be in _refold_with_ew (Lib/email/_header_value_parser.py). When a part needs encoding because it has non-ASCII characters, a non-terminal part is unpacked into its children (line 2973, "It's not a Terminal, do each piece individually"). For a Comment, the children don't include the ( and ), which only Comment.__str__ adds. The parenthesis handling added for gh-143935 (line 3036, if part.token_type == 'comment':) is only on the path for parts too long to fit on a line, not on this one. Wrapping the children the same way there (an opening ( terminal, the children, then a closing ), with the encoded words kept inside) should fix it.
I can turn these into test cases for test__header_value_parser / test_headerregistry if that helps.
If you'd like me to keep fuzzing the email package (other headers and policies, or re-checking after a fix), feel free to reach out. I'd be glad to keep working on it with you.
Thanks!
PS: Sorry if the issue looks like a template, it's, I am reporting plenty of issues to different projects and I am a simple human :)
CPython versions tested on:
3.14
Operating systems tested on:
macOS
Activity