Here I document the servers I’m running to manage the Argonaut projects, whilst defending protocols perceived to be outdated.
eMail has been with us since the dawn of the Internet, & love it or hate it (I personally love it, let me respond in my own time! Longform text! Though the protocols are ugly with basic features being bolted on later) it refuses to die.
And with projects like Maddy it’s actually quite easy to run your own. If you’re somewhat techie & want to have multiple addresses anyways, I recommend trying your hand at it!
So how does Maddy work?
After some magical infrastructure parsing commandline flags, subcommands, & other args Maddy’s main
run subcommand performs some validation, opens the specified logging output registering it with that infrastructure, if successful considers setting some debug options, configures the PATH envvar, opens & parses the config file (using a manually-written pushdown automaton, without a lexer & with a scanner supporting imports), registers logging cleanup, & runs main routine.
This main routine exracts various settings, performs some validation, registers a callback to reconfigure logging, iterates over endpoints from the config & looked up in a couple registries to get initializer routines which are immediately called (whilst validating that there’s no duplicates in the config), the resulting “endpoints” are iterated to register callbacks they’ve shutdown correctly, iterates over config to ensure everything’s been resolved, & runs a mainloop handling UNIX signals (rest of the mainloop is in the Go runtime) before triggering the shutdown event.
Core Maddy Plugins
Maddy primarily consists as a large suite of trivial “modules” extending its core mainloop & each other, implementing all the basic features eMail’s had bolted on, the alternative backends which could power each of them, & more. Today I’ll discuss the authentication modules!
It supports auth backends for PAM via C language-bindings (PAM expects C, maddy’s written in Go) & manually-parsed colon-seperated-values “shadow” file storing OS useraccounts.
Also there’s a trivial backend for plain-text username/passwords out of the configfile, or another to defer to an external command. Or to a Dovecot server via its client library, or to a NetAuth server via their client library. Or LDAP.
There’s a utility for checking the domain part of the email address we’re authenticating into matches the servers this maddy instance is running. Or an abstraction around Go’s SASL libraries to call these extensions.
But primarily there’s an authentication backend hooking up to the configured relational-database backend with an opinionated choice (ensuring old passwords can still be used) of password-encryption libraries. I don’t see any obvious issues with this crypto, & I’m glad they’re handing off the real work to dedicated modules!
There’s a plugin which runs external commands to determine how to flag an email, includes an internal class wrapping those commands & passing configured data from the emails.
There’s a suite of files declaring the HTTP APIs for various DNS providers, presumably so Maddy can autoconfigure itself with them since DNS was the least straightforward part of the install instructions. So naturally they’d want to do something about it!
There’s non-trivial rate-limitting; including bucketting, semaphores, timeouts, or some combination thereof.
There’s plugins for rewriting incoming mail, including specifically the target address, grouping, or DKIM headers.
Maddy includes a suite of checks, including:
- A trivial one enforcing use of TLS on connections
- A noop template that can easily be copied & altered
- Utility to aid implementation of stateless checks
- Validating DKIM signatures, wrapping an external library with configuration
- Checking domain names match the sender
- Validating DNS configuration (less trivial)
- Consult RSpamD via it’s HTTP API
- Wrapper around a “Milter” library checking connections, headers, & the like
- Defer to a configurable command
- Check whether the sender is authorized to email us
- Wrapper around an external SPF library adding concurrency & configurable handlers for it’s different status codes
Additionally there’s a registry of normalization functions & an email-address lookup routine for authorization.
There’s some fairly-trivial commandline infrastructure, building upon external libraries, including standard prompts, configuring stdIO, & some additional subcommands.
Most of those subcommands relate to useraccounts.
There’s a nontrivial wrapper around a DMARC library referring to DNS records, exposing a “verifier” plugin referring to “evaluate” sibling-module. Both of which are unit-tested.
And there’s a plugin to send Did-Not-Send replies as a courtesy back to the sender.
Skimming the core servers (which are started when their corresponding plugins are initialized) Maddy integrates…
There’s an optional integration of the Dovecote SASL server, quite straightforward unless I’m digging into Maddy’s dependencies. Likewise there’s integration of an IMAP server for the configured endpoints. Or Prometheus analytics wrapped in an HTTP server.
But most of its own effort goes towards implementing SMTP!
The SMTP server includes an abstraction around Go’s date & regex parsing modules to support all of SMTP’s formats. It declares some Prometheus counters. And a routine to finalize the metadata on emails being sent.
There’s a class exposing methods for the main SMTP commands, with logging & “task”-profiling, calling out to other objects with validation. More of these “directives” are defined alongside the initialization code, where they’re registered & hooked up to configured endpoints.
The actual code for parsing the directives on those sockets is managed as an external library, which also aids implementing accessor directives. (I guess I’ll have to skim @foxcpp@github’s other repos…)
This plugin does have unittests.
Maddy includes a utilities library for it's plugins to build upon, including APIs for:
- Parsing, comparing, testing,normalizing, & validating email addresses & lists thereof.
- An interface for reading from a bytes array or a file.
- A configfile handwritten parser with imports & envvar substitution.
- Another submodule which aids parsing of various fields inside that configfile, these may be more complex with their own lexer.
- DNS utilities with a custom client.
- Converting errors to a common type.
- Abstraction around a value that has not yet been computed, implemented using Go's channels & mutexes.
- Subscribable event dispatch.
- I/O abstraction for outputting logging messages (or JSON) to configured destinations, including syslog support.
- Parsing those logfiles.
- Various interfaces for the different plugins I've described to implement.
Amongst the various parser/serializer/networking libraries underlying the Maddy email server is Go-IMAP. Today I’m studying how it works.
There’s a datastructure parsing/serializing shell-like commands via an array. There’s an abstraction around mutex, with decorators including for debugging & support for “upgrading” the input/output streams with a (typically TLS) decorator. Date-parsing via standardlib with suite of templates. Parse macros & flags. Slice of a reader, a “literal”.
There’s a logging interface. A parsable/serializable/matchable mailbox datastructure. As well as a non-trivial message datastructure. A lexer parsing common tokens. Some utils for serializing IMAP responses. And the reverse. Parser for search queries. Parsing/serializing/querying sequence numbers & maintaining collections thereof. Serializing error statuses.
For submodules… There’s interfaces for backend extensions to implement, & datastructures to pass to them. With parsing-abstractions. There’s a non-persistent in-RAM backend for testing purposes. There’s a simple IMAP client library. 2 submodules implements each of the IMAP commands, each with parse, serialize, & execute methods. Or typically handling them is left to the backend. With some infrastructure to dispatch these commands. Another submodule handles serializing common responses, & possibly parsing these datastructures too. Then there’s UTF-7 transcoding.
Maddy mainly serves to implement routing emails to their recipients often using SMTP, with that being what its extensions focus on. It’s Go-IMAP dependency models the main state sync’d between email clients over the IMAP protocol, & storing this state is what its extensions focus on. Most of which includes shell commands for testing them individually. Typically the go-imap-sql backend is used hooking up to a configured relational database (e.g. SQLite), reimplemented utils, & more mess.
go-imap-sql also includes a suite of shell commands for managing this database. go-imap-mess includes some reusable template files to get started on implementing new backends, & another copy of the transient in-RAM backend. There’s a blackbox testsuite, independent of the specific backend. Some plugins extends the commands the server understands. go-imap-unselect adds the UNSELECT command to close a mailbox, lightly tweaking server state. go-imap-idle responds “done”.
go-imap-sortthread clusters emails to conversational “threads” on the server, who’s better suited to implement this efficiently (I have seen the effort clients have to go to for this…). go-imap-namespace allows switching between mailbox trees. go-imap-specialuse adds additional labels to mailboxes we want clients to treat specially. go-imap-i18nlevel configures a user-property. go-imap-appendlimit informs clients of the restriction.
go-imap-uidplus exposes additional fields, without implementing the logic itself. go-imap-enable provides syntax for turning on specified server notifications. go-imap-maildir stores emails in the filesystem, using the classic defacto standards.
When the iPhone came out we lost good tools who didn’t respond fast enough to the disruption. Computing most certainly wasn’t better back then, but it did feel like we had a better vision.
XMPP was in a fragile state with few end-users being aware of how ubiquitously it powered most instant messengers of the time. Especially with them refusing to federate (in no way the fault of these standards!).
Despite hearsay from those who haven’t tried it, today XMPP is a painless IM protocol.
XMPP (despite its incorrect reputation) is a full-featured mobile-friendly lightweight encrypted federated instant messaging protocol based on streaming XML over (amongst others) TLS. Once I setup a @snikket_im server for my #ArgonautStack , it has been extremely painless. Even when I tested federation against a personal YUNoHost server, which I hear good things about.
Under the hood Snikket’s server is a branded Docker packaging of Prosody, so today I’ll start studying Prosody’s core.
Prosody includes a startup script (which I’ll largely defer until a later thread) that sanitizes its input, initializes numerous components, & runs a mainloop with error-handling. As well as a nice commandline interface to edit & reload various pieces of configuration.
There’s a collection of connections to other XMPP servers, with logic to initialize these connections & tear them down. There’s component which routes XML elements to the appropriate client, peer-server, or internal plugin.
That routing is spread out across 2 files, one to interpret the elements & one to manage the destinations. Yet another file imiplements routing to/from hosts.
There’s an abstraction around configuration globals. There’s some simple logging infrastructure. There’s a collection of cryptographic certificates. There’s a collection of ports its listening on. There’s statistics-gathering. There’s collections of contacts & subscriptions. There’s a collection of persistence backends.
There’s a collection of modules & (in a separate file) a public API for them. And finally a collection of user-accounts, deferring to their “hosts” for authentication.
As a (chat) server, around the data-routing core I described last time, a core job of Prosody is to implement networking! Today I’m exploring its “net” subsystem.
This includes a global “cqueue” (importing that titular module) adding timeout/file trigger conditions. They’ve replaced a deprecated HTTP API with an error message. There’s a collection of open connections.
These are all nicely sandboxed thanks to Lua (the elegant scripting language its implemented in) making that easy.
There’s a choice of configurable mainloops, repeatedly taking action for the next socket to receive data.
There’s a WebSockets channel to support in-browser clients. There’s a channel wrapping LibUnbound. There’s a hack “STUN” to splice together clients’ UDP streams overcoming IPv4 (why aren’t we on 6!!) limitations felt in videocalling. There’s a cached DNS client, & an asynchronous one, with its subsystem including utilities to concate, map-over, or create iterators over query results.
There’s an HTTP client & (in a subsubsystem) server. There’s 3 different backends for the mainloop including TLS & abstractions to hook on these servers.
That HTTP server includes a fileserver, error-code mappings, manually parses requests, & a couple other things.
There’s a submodule splitting off some of the WebSockets implementation.
Despite being written very-much from scratch, this Lua code looks very simple!
Prosody largely consists of a large suite of extensions around its minimal core. It’ll take a few days to go over all of these, but starting with the more miscellanea ones I see:
- Replying with the current time
- Communicating which STUN & TURN(S) servers it is running for clients to use
- Track deleted accounts to share their status & prevent username reuse
- Track uptime
- Token-based authorization
- Download VCards (a few variants thereof)
- User-registration over XMPP
- Report version numbers
- Notify admins of new accounts
- Show a welcome message to new users
- Server-to-server certificate checks, reporting to logfile
- Enable bidirectional streaming
- Report contact addresses for abuse-reporting
- Filter out empty messages
- Shakespearean test data
- Storage backends including noop, in-memory, SQL (complex), or XML files
- Load appropriate registration plugins
- Query the private storage
- TLS message-transport channel
- Websocket channel
- Manage “rosters” of contacts, removing deleted accounts
- Enforce registration limits/throttling
- SOCKS5 proxying
- Another plugin for registering accounts over XMPP
- Respond to pings
- Respond with a unique name
- Track the protocols Prosody exposes
- Send configured “Messages of the Day”
- Track & report last activity for each individual user
- Legacy unhashed authentication
- Query your open sessions
- Extend routing to include groups, headlines, etc
- Enforce limits
- Invite users to create accounts, with related plugin so registered users can invite others & for accepting those invites
- Prevent usernames from being too similar
- Integration into OpenMetrics
- Report HTTP errors
- Distribute attachments over HTTP
- Manage groups within rosters
- POSIX daemonisation
- More tracking of services & their credentials
- Enable SQL query logging
- Dialback server-to-server authentication
- Server-to-server discovery
- HTTP access control & routing
- Route CSI events, with a simple implementation prioritizing messages to forward on
- Register cronjobs
- Publish/subscribe channels
- Allow admins to create invites
- Archiving data
- Account discovery
- Track user presence
- Manage roles
- Insecure authentication (this plugin is heavily discouraged)
- Propagate admin announcements
- Anymous server-to-server datastorage
- Authentication with hashed passwords, another for unhashed passwords
- Allow admins to register accounts
- A REPL interface
- UNIX-socket wrapping other connections
- TELNET interface to that REPL
- LDAP authentication
- Manage blocklists
- Manage bookmarks
- Stream messages over hacky HTTP (so we could have XMPP webclients before WebSockets)
- Client-to-server over the streaming container-protocols
- Copy messages to be sent to specified others
- Parse the XML streams
- HTTP filesharing
- Server-to-server protocol
- SASL authentication
- Datastorage deferring to a “self” object
- Error reporting
These are all mostly mutually-independent, and individually quite simple if not trivial. Especially the earlier ones I listed.
More may be added purely client-side, & I’ll continue describing more over the following days.
Looking at a slightly more complex Prosody-plugin today (I’ll continue with others over the next couple days), I’m looking at one which maintains a global “commands” mapping, for one client to upload values to & others to download values from.
Upon download there’s access-control, state tracking, & error-handling. With a supporting library for the latter 2.
Skimming the next less-trivial Prosody built-in plugin, there’s a Message-Archive-Management system which allows XMPP clients to configure & enact auto-archiving of messages (is this for the sake of spam?).
It includes some trivial support modules to store this configuration in “sessions” & convert from the uploaded XML.
This includes plenty of validation, & only applies to the user making these requests.
Looking at the next less-trivial Prosody plugin, there’s a publish-subscribe communication primitive, with builtin auto-archiving & affiliations.
It defers to a support module adding checks around a utility library I’ll explore later…
Finishing my Prosody plugins… There’s a Multi-User-Chat (MUC) suite!
- Locking & unlocking rooms
- Managing official languages
- Configuring a “moderator” granting permission to talk
- Configuring room name
- Configure whether to delete a room when empty
- Password-protected rooms
- View attendees in a room
- A support library tracking room participants & their roles, or reporting errors
- Configure room “subject” description
- Something relating to messages towards clients
- Configure room “description” (what’s the difference from subject?)
- Validates “hats” claimed by clients
- Configure whether to list the room publicly
- Configure whether to allow the public to access this room, & allow inviting others to join it
- Support library to manage occupants & their sessions
- Support library to track occupant IDs
- Broadcast occupants’ presence within a room
- Requests to speak over audio/video chat
- Manage & enforce registrations
- Track & expose room history
- More moderation options
- And some general infrastructure for data-modelling rooms & routing messages within them
Continuing my studies of Prosody, the XMPP-server bundles a handful of “tools” including:
- Parsing an XML file as an XMPP “stanza” & outputs the result. Testing the serialization & parsing support code with a bit of intermediate-processing.
- Various data to test the formatting & possibly serialization support code on.
- Parse the XML file of HTTP status codes provided by IANA & convert it into Lua code.
- Callback to improve Lua’s error reporting.
- A script to clone https://hg.prosody.im/prosody-modules/ & generate buildfiles.
- A script to reorganize the how this code is organized on the filesystem.
- Various data & code to upgrade database to latest schema.
- Debugging utilities wrapping specified functions with logging.
- Convert IANA’s XML file describing DNS parameters to Lua code.
- Parse Erlang data.
- Parse OpenFire’s XML database to migrate into Prosody.
- Output configuration into a legible format.
- Parse eJabberd’s Erlang & database files to migrate into Prosody.
- Parse Jabberd’s SQL (with the help of a parser-generator) to migrate into Prosody.
The Prosody XMPP server I’ve been studying is written in Lua. Lua proves to be very expressive (and fast) for Prosody’s purposes, but it does have a smaller standard lib. And a smaller ecosystem.
As such Prosody bundles several utility libs for its own use! Including (but not limited to, I will explore more later):
- Throttling calls to a given rate
- Randomly generate UUIDs
- A priority heap scheduling timers, to be hooked into the mainloop
- Rendering coloured text to terminals or HTML
- Some sort of meta-programming magic indexing parsed XMPP-stanzas
- Watchdog timer checking whether the mainloop’s responsive
- A more optimized queue
- Wrapper around Lua XML Parser (LXP) to handle namespaces & simplify callbacks, dispatching to the “stanza” module
- regexp & evaluate simple template strings, boolean expressions, & externally-provided “filter” callbacks yielding an output string from XML
- Read /dev/urandom
- Dispatch to appropriate SASL authentication mechanism
- XML-serialize message archives
- Logged & filtered wrapper around writing to sockets
- “Promises” of future results, much like the JS recently-builtin class.
- Locate resource with greatest priority
- Underlying ringbuffer queue
- Sort destinations by preference
- Configuration storage
- Gathering statistics
- Histogram-analyzing statistics
- Various certificate checks
- Check mercurial repo
- Simple logger
- Signing JSON (JWT)
- Running plugins
- Processing filepaths
- Serialize Lua’s datamodel
- Set operations
- Nicer SQL-database APIs
- Output XMPP stanzas
- Wrap Lua imports with caching
- Resolve a JSON-Path
- Basic JSON schema validator
- OpenSSL bindings via shell commands
- Parse VCards
- Serialize hexcodes
- Format common logging messages
- A “hashring”
- Configure the garbage collector
- Generate links with a certain number of random chars
- URL processing
- Binary heap collections
- Parse/interpret format strings
- Another templating language
- Maintain a collection of filters
- Processing Jabber-IDs
- Merge multiple “tables” (Lua’s only collection type)
- Subcommands for
- Date formatting
- Registry of DNS codes
- Envvar utils, supporting older Lua versions
- JSON parser
- IP address processing
- Subscribable events
- OpenMetrics client
- Error reporting
- DNS processing
- Bootstrap a running Prosody daemon
- Build a Stanza object
- Build “data forms”
- Parse commandline flags, across different Lua versions
- Bitwise processing
- Asyncronous processing wrapping coroutines
- Check dependencies
- Simple data conversions
- Dynamic buffers throttling I/O
- Track server data in-memory
- Richer pubsub infrastructure
- Datamodel for “data forms”
- List processing
- Serialize XML schema for admins
I’d have split some of these off to indicate that they’re more central to XMPP…
Prosody Terminal Utils
Prosody bundles a handful of utilities to help it build a nicer (if still textual) user interface, including:
- Reading lines or (via
stty raw -echo 2>/dev/null) characters
- Formatting units to indicate a number’s order-of-magnitude, base 2 or 10
- Reading a password without echoing
- Prompt for a yes or no
- Prompt for a new, double-checked password
- Output a prompt
- Output a formatted string
- Pad text with spaces
- Truncate text, with a wrapper adding ellipses
- Layout/format a table
In its utility suite the Prosody XMPP Server includes some subcommands for its
prosodyctl command. Amongst the less trivial ones (which don’t fit in the main file for these subcommands) are:
- A Read-Eval-Print Loop (REPL) for running these subcommands within a dedicated shell
- Various validation checks
- Loading & server certificates
All including user interaction via the terminal.
Prosody SASL Authenticators
Prosody’s SASL backend (organized within its “util” junkbox) includes a handful of authentication methods:
- Plain compares passwords as plaintext (discouraged!)
- External runs a callback function
- Salted-hashed passwords to store as strings
- Connect as anonymous user
Prosody C Code
Looking at the supporting utils implemented in C, there’s:
- Bindings to
- Table-initialization utils (tables being Lua’s only collection type)
- Bindings to Windows’ DNS & console APIs
- Networking API bindings
xpcallto older Lua versions
- Random-Number Generation bindings
- Polyfill bitwise operations
- Bindings to various hash functions
- Binding UNIX signals
- Interpret format strings to encode & decode binary data
- Bindings to
ppollsyscall & equivalents
- Faster ringbuffer
- Encode binary data as text
- Additional POSIX bindings