Inline Comments in dotenv and EnvironmentFile Parsers Break Credentials Quietly

A surprising number of production incidents trace back to a single line in a config file that looks completely correct to a human reader:

API_TOKEN=abc123def456 # rotated 2025-01-14

The intent is obvious. The value is abc123def456. The # rotated 2025-01-14 is a note for the next person who opens the file. Almost every developer who writes that line believes the comment is inert.

It is not. Whether the trailing text becomes part of the value depends entirely on which parser reads the file, and the parsers disagree. The result is a credential that is almost right — correct prefix, correct length range, wrong suffix — which is the worst possible failure mode. Authentication fails, but the error message points at the token, not at the file.

Two families of parsers, two different rules

Environment files are read by at least two distinct lineages of software, and they were never specified together.

The shell lineage. systemd's EnvironmentFile=, shell source, and POSIX-style .env loaders inherit shell word-splitting semantics. In a shell, # starts a comment only when it appears at the beginning of a word. Inside an unquoted assignment, behavior is subtle and varies between shells.

The application lineage. Language-level dotenv libraries — the kind loaded by Node, Python, Ruby, Go, and PHP applications — implement their own tokenizers. Many of them treat # as a comment introducer anywhere on the line, but only when preceded by whitespace, and only when the value is unquoted. Some strip it unconditionally. Some do not strip it at all.

The consequence: the same file can produce different values in systemd and in the application that consumes the environment. A deployment that works when the process is started manually can fail when it is started by the init system, or vice versa.

A concrete divergence

Consider this file:

# .env
DATABASE_URL=postgres://app:hunter2@db.internal:5432/app # primary
SMTP_PASSWORD=correct horse battery staple # do not rotate without ops

A parser that strips inline comments yields:

DATABASE_URL=postgres://app:hunter2@db.internal:5432/app
SMTP_PASSWORD=correct horse battery staple

A parser that does not strip them yields:

DATABASE_URL=postgres://app:hunter2@db.internal:5432/app # primary
SMTP_PASSWORD=correct horse battery staple # do not rotate without ops

The second value is not a password. It is a password followed by a space, a hash, and a sentence. The database driver will happily attempt to connect with it, fail authentication, and return a generic error. Nothing in the stack says "your password contains a comment."

Why the failure is silent

Secret-loading code is usually written to be tolerant. It reads a string, hands it to a client library, and lets the client library decide what to do. Client libraries are also tolerant: they accept any string as a credential and only discover the problem at the network boundary.

By then, the error has been transformed several times:

1. The parser produced a string with extra characters.
2. The client library accepted it without validation.
3. The remote service rejected it with a generic 401 or 403.
4. The application logged "authentication failed" without the credential value.

The trail from symptom to cause is broken at step 1, and step 1 is invisible because the file looks correct. This is why inline comments are a recurring source of broken credentials rather than a one-time gotcha.

The quoting trap

Quoting does not reliably solve the problem, because parsers disagree about what quotes mean too.

# Three ways to write the same intent
A=secret # comment
B="secret" # comment
C='secret # not a comment'

• A may or may not include # comment.
• B may or may not include # comment after the closing quote.
• C is explicitly a value containing a hash, which is correct if that is what you meant, and wrong if you meant a comment.

Some parsers strip everything after the first unquoted #. Some strip only if the # is preceded by whitespace. Some treat # inside single quotes as literal and inside double quotes as literal, but then fail to handle escaped quotes. There is no single rule that all implementations follow.

What teams typically do about it

The pattern that holds up across parsers is boring: do not put comments on the same line as a value. Put them on their own line, above the key.

# primary database, rotated 2025-01-14
DATABASE_URL=postgres://app:hunter2@db.internal:5432/app

# do not rotate without ops
SMTP_PASSWORD=correct horse battery staple

This is portable. Every parser that understands KEY=value also understands a full-line comment. The cost is vertical space, which is cheap. The benefit is that the file means the same thing everywhere.

A second pattern is to validate at load time rather than at use time. If a value is supposed to be a token, check its shape before handing it to a client:

import os
import re

TOKEN_RE = re.compile(r"^[A-Za-z0-9_\-]{20,}$")

def load_token(name: str) -> str:
    raw = os.environ.get(name)
    if raw is None:
        raise RuntimeError(f"{name} is not set")
    value = raw.strip()
    if not TOKEN_RE.fullmatch(value):
        # Do not log the value; log its shape.
        raise RuntimeError(
            f"{name} does not look like a token "
            f"(length={len(value)}, has_space={' ' in value}, "
            f"has_hash={'#' in value})"
        )
    return value

The error message deliberately reports properties of the value rather than the value itself. has_hash=True is enough to point an operator at the file. Logging the raw value would leak the credential into whatever collects logs.

Making the parser fail loudly

Tolerance is the enemy here. A loader that silently accepts a malformed line converts a configuration error into a runtime authentication error, which is harder to diagnose and more expensive to debug.

A stricter loader does three things:

1. Rejects trailing content after an unquoted value. If the line contains a # that is not at the start, and the value is not quoted, raise an error rather than guessing.
2. Rejects values containing whitespace unless quoted. A password with a space is legal but rare; requiring quotes makes the intent explicit.
3. Reports the line number and key name, never the value. The operator needs to find the line, not read the secret.

def parse_env_line(line: str, lineno: int) -> tuple[str, str] | None:
    stripped = line.strip()
    if not stripped or stripped.startswith("#"):
        return None
    if "=" not in stripped:
        raise ValueError(f"line {lineno}: missing '='")
    key, _, raw = stripped.partition("=")
    key = key.strip()
    raw = raw.strip()
    if raw.startswith('"') and raw.endswith('"') and len(raw) >= 2:
        return key, raw[1:-1]
    if raw.startswith("'") and raw.endswith("'") and len(raw) >= 2:
        return key, raw[1:-1]
    if "#" in raw:
        raise ValueError(
            f"line {lineno}: unquoted value for {key!r} contains '#'; "
            "move the comment to its own line or quote the value"
        )
    if any(c.isspace() for c in raw):
        raise ValueError(
            f"line {lineno}: unquoted value for {key!r} contains whitespace; "
            "quote it if that is intentional"
        )
    return key, raw

This parser refuses to guess. It will reject files that other parsers accept, which is the point: the rejection happens at load time, in a place where the operator can see it, instead of at authentication time, in a place where the error is opaque.

The systemd-specific wrinkle

systemd's EnvironmentFile= has its own rules, and they are not identical to any dotenv library. In particular, the handling of quotes, escapes, and trailing whitespace differs. A file that a Python dotenv loader reads one way may be read differently by the unit that starts the process.

A common pattern in teams that run both is to keep two files: one for the application loader and one for the unit file, generated from a single source of truth. Duplication is unpleasant, but it is less unpleasant than a credential that works in one path and not the other.

A lighter alternative is to avoid EnvironmentFile= for secrets entirely and inject them through a credential mechanism that does not involve parsing a text file at all. That removes the class of bug rather than working around it.

Why this keeps happening

The root cause is not that any one parser is wrong. It is that the format was never standardized. .env files are a convention, not a specification, and every implementation filled in the ambiguous cases according to its own judgment. Inline comments are exactly the kind of ambiguity that different authors resolve differently.

The practical response is to treat the format as if it were strict even when the parser is lenient:

• Comments on their own lines, always.
• Values quoted when they contain anything other than [A-Za-z0-9_.\-].
• Loaders that validate shape and fail loudly.
• Error messages that describe the value's shape, never the value.

None of this is clever. It is the kind of discipline that prevents a class of incident where the file looks fine, the logs look fine, and the only signal is a 401 from a service that will not say why.

A short checklist

Before the next deploy, it is worth grepping for the pattern:

grep -nE '^[A-Za-z_][A-Za-z0-9_]*=.*#' .env .env.* 2>/dev/null

Any hit is a candidate for the bug described here. The fix is mechanical: move the comment to its own line. The harder part is the second half — making the loader refuse to accept the ambiguous form in the first place, so the next person who writes KEY=value # note gets an error at load time instead of a mystery at authentication time.

Credentials fail in many ways. This one is unusual because the file is not wrong in any obvious sense. It is wrong only relative to a parser that the author did not have in mind. Making that assumption explicit — in the file, in the loader, and in the error message — is the difference between a five-minute fix and an afternoon of chasing a 401.