rmail — Architecture

Package Structure

rmail/
├── main.py              Server entrypoint, lifecycle, signal handling
├── models.py            Dataclasses: ServerConfig, TLSConfig, RelayConfig, User, MessageMeta
├── config.py            JSON config loading, validation, default generation
├── storage.py           Flag encoding, filename parsing, shard calculation
├── auth.py              User management, PBKDF2-SHA256 hashing
├── aliases.py           Per-domain regex alias routing
├── delivery.py          Multi-worker async delivery queue (local + relay routing)
├── rate_limiter.py      Per-IP sliding-window rate limiting
├── validate.py          Input validators: String, Email, Password, Domain, Username
├── http.py              HTTP/1.1 request/response parser
├── tls.py               Auto-generated TLS certificate via openssl
├── backend/
│   ├── base.py          MailStoreBackend ABC, QuotaExceededError
│   └── filesystem.py    Shard-based filesystem implementation
├── smtp/
│   ├── address.py       Address extraction from MAIL FROM / RCPT TO
│   ├── session.py       SMTP state machine: EHLO, AUTH, STARTTLS, relay gating
│   └── server.py        Asyncio listener, connection limits, TLS, delivery wiring
├── imap/
│   ├── constants.py     State constants, SPECIAL-USE attribute map
│   ├── session.py       Core session, authentication, capability, command dispatch
│   ├── mailbox_cmds.py  SELECT, EXAMINE, LIST, LSUB, STATUS, IDLE, RENAME, etc.
│   ├── message_cmds.py  FETCH, STORE, SEARCH, APPEND, COPY, MOVE, EXPUNGE, etc.
│   ├── parse.py         IMAP command and argument parsing
│   ├── search.py        SEARCH criteria evaluation engine
│   ├── message.py       RFC822 message parsing helpers
│   ├── io.py            Binary literal I/O helpers
│   └── server.py        Asyncio listener, TLS, connection semaphore
├── relay/
│   ├── mx.py            Async DNS MX resolver: raw UDP packets, pointer compression, TTL cache
│   ├── client.py        Async SMTP client: STARTTLS, AUTH PLAIN, dot-stuffing
│   └── queue.py         Relay queue: async workers, retry schedule, DSN bounce delivery
├── api/
│   ├── server.py        HTTP routing, JWT auth middleware, CORS, lifecycle
│   ├── handlers.py      All request handler methods (_APIHandlersMixin)
│   ├── token.py         JWT generation and validation (base64 + HMAC-SHA256)
│   └── autoconfig.py    Thunderbird autoconfig and Exchange autodiscover XML
└── cli/
    ├── commands.py      init, alias, migrate, install, uninstall
    ├── diagnose.py      8-phase diagnostic suite
    ├── test.py          Integration test suite
    └── ports.py         Port management, kill conflicting processes

Inbound Mail Flow

An external sender delivers a message to a local user:

TCP connection accepted by SMTPServer, connection semaphore checked
Rate limiter checks per-IP connection rate; locked-out IPs are rejected immediately
SMTPSession drives the state machine: GREETING → GREETED → MAIL → RCPT → DATA
RCPT TO validation: domain checked against configured domains; local user or alias looked up; unauthenticated external recipients rejected
DATA received, size enforced; message assembled
DeliveryQueue.enqueue() receives recipients and raw message; returns immediately (non-blocking)
Delivery worker splits recipients into local and external; calls backend.append_message() for local, relay_queue.enqueue() for external
FileMailStoreBackend allocates UID via fcntl.flock(), writes {uid:08d}_.eml into the appropriate shard directory

Outbound Relay Flow

An authenticated user sends to an external domain:

RCPT TO on SMTPSession: domain not local, user authenticated, relay enabled → accepted
DeliveryQueue routes to RelayQueue.enqueue() with mail_from, rcpt_to, raw message
Relay worker determines delivery path: smarthost (if configured) or direct MX
MXResolver.resolve() performs raw DNS UDP query for MX records, sorts by preference, caches by TTL; falls back to domain-as-host if no MX records found
relay_message() connects via asyncio.open_connection(), issues EHLO, attempts STARTTLS (opportunistic), authenticates if smarthost credentials provided, transmits with dot-stuffing
On failure: transient → added to in-memory retry list with scheduled retry time; permanent (5xx) → DSN bounce written to sender's INBOX via backend
Retry scheduler wakes every 30s, re-enqueues messages past their retry time

IMAP Session Flow

IMAPServer accepts connection, sends capability greeting
IMAPSession dispatches commands via mixin pattern: _MailboxCommandsMixin and _MessageCommandsMixin
Authentication: LOGIN or AUTHENTICATE PLAIN; user loaded via auth.authenticate()
SELECT/EXAMINE: scans shard directories via os.scandir(), builds message list, reports EXISTS/RECENT/UNSEEN
FETCH: reads individual .eml file; computes ENVELOPE, BODY structure on demand
STORE: renames file to encode new flags; uses compare-and-swap retry on conflict
IDLE: session polls backend at 10s intervals, sends * N EXISTS when count changes

Storage Model

Each message is a single .eml file. Flags are encoded in the filename, eliminating all index writes on flag changes.

mailstore/domains/example.com/users/alice/mailboxes/INBOX/
├── status.json          {"uid_validity": 1700000000, "uid_next": 42}
├── 0000/
│   ├── 00000001_.eml    no flags
│   ├── 00000002_S.eml   \Seen
│   └── 00000041_AFS.eml \Answered \Flagged \Seen
└── 0001/
    └── 00001000_SF.eml  \Seen \Flagged

Flag characters (alphabetically sorted in filename): A=\Answered, D=\Deleted, F=\Flagged, S=\Seen, T=\Draft.

Shard directory = uid // 1000, zero-padded to 4 digits. Maximum 1000 files per directory.

Operation complexity

Operation	Complexity	Lock
Append message	O(1)	fcntl.flock (UID alloc only)
Read message	O(1)	None
Modify flags	O(1) os.rename	None (CAS retry on conflict)
Delete message	O(1) os.unlink	None
List messages	O(n) scandir	None
IMAP SELECT	O(n) scandir + status.json read	None

Key Design Patterns

Mixin composition — IMAP session split into _MailboxCommandsMixin and _MessageCommandsMixin; Exchange API split into ExchangeAPI and _APIHandlersMixin. Keeps files under ~400 lines each.
Async worker queues — DeliveryQueue and RelayQueue follow identical patterns: asyncio.Queue, SENTINEL shutdown, configurable worker count.
Abstract backend — MailStoreBackend ABC defines the storage interface; FileMailStoreBackend is the only implementation. Alternative backends can be added without touching protocol code.
Dataclass configuration — all config is typed via dataclasses (ServerConfig, SMTPConfig, RelayConfig, etc.) with validation at load time.
Lock-free concurrency — flag changes use os.rename() which is atomic on POSIX. Only UID allocation requires a lock (sub-millisecond hold time).