Project Management

Here I document the servers I’m running to manage the Argonaut projects, whilst defending protocols perceived to be outdated.

Maddy

eMail has been with us since the dawn of the Internet, & love it or hate it (I personally love it, let me respond in my own time! Longform text! Though the protocols are ugly with basic features being bolted on later) it refuses to die.

And with projects like Maddy it’s actually quite easy to run your own. If you’re somewhat techie & want to have multiple addresses anyways, I recommend trying your hand at it!

So how does Maddy work?

After some magical infrastructure parsing commandline flags, subcommands, & other args Maddy’s main run subcommand performs some validation, opens the specified logging output registering it with that infrastructure, if successful considers setting some debug options, configures the PATH envvar, opens & parses the config file (using a manually-written pushdown automaton, without a lexer & with a scanner supporting imports), registers logging cleanup, & runs main routine.

This main routine exracts various settings, performs some validation, registers a callback to reconfigure logging, iterates over endpoints from the config & looked up in a couple registries to get initializer routines which are immediately called (whilst validating that there’s no duplicates in the config), the resulting “endpoints” are iterated to register callbacks they’ve shutdown correctly, iterates over config to ensure everything’s been resolved, & runs a mainloop handling UNIX signals (rest of the mainloop is in the Go runtime) before triggering the shutdown event.

Core Maddy Plugins

Maddy primarily consists as a large suite of trivial “modules” extending its core mainloop & each other, implementing all the basic features eMail’s had bolted on, the alternative backends which could power each of them, & more. Today I’ll discuss the authentication modules!

It supports auth backends for PAM via C language-bindings (PAM expects C, maddy’s written in Go) & manually-parsed colon-seperated-values “shadow” file storing OS useraccounts.

Also there’s a trivial backend for plain-text username/passwords out of the configfile, or another to defer to an external command. Or to a Dovecot server via its client library, or to a NetAuth server via their client library. Or LDAP.

There’s a utility for checking the domain part of the email address we’re authenticating into matches the servers this maddy instance is running. Or an abstraction around Go’s SASL libraries to call these extensions.

But primarily there’s an authentication backend hooking up to the configured relational-database backend with an opinionated choice (ensuring old passwords can still be used) of password-encryption libraries. I don’t see any obvious issues with this crypto, & I’m glad they’re handing off the real work to dedicated modules!

There’s a plugin which runs external commands to determine how to flag an email, includes an internal class wrapping those commands & passing configured data from the emails.

There’s a suite of files declaring the HTTP APIs for various DNS providers, presumably so Maddy can autoconfigure itself with them since DNS was the least straightforward part of the install instructions. So naturally they’d want to do something about it!

There’s non-trivial rate-limitting; including bucketting, semaphores, timeouts, or some combination thereof.

There’s plugins for rewriting incoming mail, including specifically the target address, grouping, or DKIM headers.

Maddy checks

Maddy includes a suite of checks, including:

Additionally there’s a registry of normalization functions & an email-address lookup routine for authorization.

Maddy Commandline

There’s some fairly-trivial commandline infrastructure, building upon external libraries, including standard prompts, configuring stdIO, & some additional subcommands.

Most of those subcommands relate to useraccounts.

There’s a nontrivial wrapper around a DMARC library referring to DNS records, exposing a “verifier” plugin referring to “evaluate” sibling-module. Both of which are unit-tested.

And there’s a plugin to send Did-Not-Send replies as a courtesy back to the sender.

Maddy Servers

Skimming the core servers (which are started when their corresponding plugins are initialized) Maddy integrates…

There’s an optional integration of the Dovecote SASL server, quite straightforward unless I’m digging into Maddy’s dependencies. Likewise there’s integration of an IMAP server for the configured endpoints. Or Prometheus analytics wrapped in an HTTP server.

But most of its own effort goes towards implementing SMTP!

The SMTP server includes an abstraction around Go’s date & regex parsing modules to support all of SMTP’s formats. It declares some Prometheus counters. And a routine to finalize the metadata on emails being sent.

There’s a class exposing methods for the main SMTP commands, with logging & “task”-profiling, calling out to other objects with validation. More of these “directives” are defined alongside the initialization code, where they’re registered & hooked up to configured endpoints.

The actual code for parsing the directives on those sockets is managed as an external library, which also aids implementing accessor directives. (I guess I’ll have to skim @foxcpp@github’s other repos…)

This plugin does have unittests.

Maddy Utilities

Maddy includes a utilities library for it's plugins to build upon, including APIs for:

Go-IMAP

Amongst the various parser/serializer/networking libraries underlying the Maddy email server is Go-IMAP. Today I’m studying how it works.

There’s a datastructure parsing/serializing shell-like commands via an array. There’s an abstraction around mutex, with decorators including for debugging & support for “upgrading” the input/output streams with a (typically TLS) decorator. Date-parsing via standardlib with suite of templates. Parse macros & flags. Slice of a reader, a “literal”.

There’s a logging interface. A parsable/serializable/matchable mailbox datastructure. As well as a non-trivial message datastructure. A lexer parsing common tokens. Some utils for serializing IMAP responses. And the reverse. Parser for search queries. Parsing/serializing/querying sequence numbers & maintaining collections thereof. Serializing error statuses.

For submodules… There’s interfaces for backend extensions to implement, & datastructures to pass to them. With parsing-abstractions. There’s a non-persistent in-RAM backend for testing purposes. There’s a simple IMAP client library. 2 submodules implements each of the IMAP commands, each with parse, serialize, & execute methods. Or typically handling them is left to the backend. With some infrastructure to dispatch these commands. Another submodule handles serializing common responses, & possibly parsing these datastructures too. Then there’s UTF-7 transcoding.

Maddy mainly serves to implement routing emails to their recipients often using SMTP, with that being what its extensions focus on. It’s Go-IMAP dependency models the main state sync’d between email clients over the IMAP protocol, & storing this state is what its extensions focus on. Most of which includes shell commands for testing them individually. Typically the go-imap-sql backend is used hooking up to a configured relational database (e.g. SQLite), reimplemented utils, & more mess.

go-imap-sql also includes a suite of shell commands for managing this database. go-imap-mess includes some reusable template files to get started on implementing new backends, & another copy of the transient in-RAM backend. There’s a blackbox testsuite, independent of the specific backend. Some plugins extends the commands the server understands. go-imap-unselect adds the UNSELECT command to close a mailbox, lightly tweaking server state. go-imap-idle responds “done”.

go-imap-sortthread clusters emails to conversational “threads” on the server, who’s better suited to implement this efficiently (I have seen the effort clients have to go to for this…). go-imap-namespace allows switching between mailbox trees. go-imap-specialuse adds additional labels to mailboxes we want clients to treat specially. go-imap-i18nlevel configures a user-property. go-imap-appendlimit informs clients of the restriction.

go-imap-uidplus exposes additional fields, without implementing the logic itself. go-imap-enable provides syntax for turning on specified server notifications. go-imap-maildir stores emails in the filesystem, using the classic defacto standards.

Prosody

When the iPhone came out we lost good tools who didn’t respond fast enough to the disruption. Computing most certainly wasn’t better back then, but it did feel like we had a better vision.

XMPP was in a fragile state with few end-users being aware of how ubiquitously it powered most instant messengers of the time. Especially with them refusing to federate (in no way the fault of these standards!).

Despite hearsay from those who haven’t tried it, today XMPP is a painless IM protocol.

XMPP (despite its incorrect reputation) is a full-featured mobile-friendly lightweight encrypted federated instant messaging protocol based on streaming XML over (amongst others) TLS. Once I setup a @snikket_im server for my #ArgonautStack , it has been extremely painless. Even when I tested federation against a personal YUNoHost server, which I hear good things about.

Under the hood Snikket’s server is a branded Docker packaging of Prosody, so today I’ll start studying Prosody’s core.

Prosody includes a startup script (which I’ll largely defer until a later thread) that sanitizes its input, initializes numerous components, & runs a mainloop with error-handling. As well as a nice commandline interface to edit & reload various pieces of configuration.

There’s a collection of connections to other XMPP servers, with logic to initialize these connections & tear them down. There’s component which routes XML elements to the appropriate client, peer-server, or internal plugin.

That routing is spread out across 2 files, one to interpret the elements & one to manage the destinations. Yet another file imiplements routing to/from hosts.

There’s an abstraction around configuration globals. There’s some simple logging infrastructure. There’s a collection of cryptographic certificates. There’s a collection of ports its listening on. There’s statistics-gathering. There’s collections of contacts & subscriptions. There’s a collection of persistence backends.

There’s a collection of modules & (in a separate file) a public API for them. And finally a collection of user-accounts, deferring to their “hosts” for authentication.

Prosody Networking

As a (chat) server, around the data-routing core I described last time, a core job of Prosody is to implement networking! Today I’m exploring its “net” subsystem.

This includes a global “cqueue” (importing that titular module) adding timeout/file trigger conditions. They’ve replaced a deprecated HTTP API with an error message. There’s a collection of open connections.

These are all nicely sandboxed thanks to Lua (the elegant scripting language its implemented in) making that easy.

There’s a choice of configurable mainloops, repeatedly taking action for the next socket to receive data.

There’s a WebSockets channel to support in-browser clients. There’s a channel wrapping LibUnbound. There’s a hack “STUN” to splice together clients’ UDP streams overcoming IPv4 (why aren’t we on 6!!) limitations felt in videocalling. There’s a cached DNS client, & an asynchronous one, with its subsystem including utilities to concate, map-over, or create iterators over query results.

There’s an HTTP client & (in a subsubsystem) server. There’s 3 different backends for the mainloop including TLS & abstractions to hook on these servers.

That HTTP server includes a fileserver, error-code mappings, manually parses requests, & a couple other things.

There’s a submodule splitting off some of the WebSockets implementation.

Despite being written very-much from scratch, this Lua code looks very simple!

Prosody Plugins

Prosody largely consists of a large suite of extensions around its minimal core. It’ll take a few days to go over all of these, but starting with the more miscellanea ones I see:

These are all mostly mutually-independent, and individually quite simple if not trivial. Especially the earlier ones I listed.

More may be added purely client-side, & I’ll continue describing more over the following days.

Looking at a slightly more complex Prosody-plugin today (I’ll continue with others over the next couple days), I’m looking at one which maintains a global “commands” mapping, for one client to upload values to & others to download values from.

Upon download there’s access-control, state tracking, & error-handling. With a supporting library for the latter 2.

Skimming the next less-trivial Prosody built-in plugin, there’s a Message-Archive-Management system which allows XMPP clients to configure & enact auto-archiving of messages (is this for the sake of spam?).

It includes some trivial support modules to store this configuration in “sessions” & convert from the uploaded XML.

This includes plenty of validation, & only applies to the user making these requests.

Looking at the next less-trivial Prosody plugin, there’s a publish-subscribe communication primitive, with builtin auto-archiving & affiliations.

It defers to a support module adding checks around a utility library I’ll explore later…

Finishing my Prosody plugins… There’s a Multi-User-Chat (MUC) suite!

Including:

Prosody Tooling

Continuing my studies of Prosody, the XMPP-server bundles a handful of “tools” including:

Prosody Utils

The Prosody XMPP server I’ve been studying is written in Lua. Lua proves to be very expressive (and fast) for Prosody’s purposes, but it does have a smaller standard lib. And a smaller ecosystem.

As such Prosody bundles several utility libs for its own use! Including (but not limited to, I will explore more later):

I’d have split some of these off to indicate that they’re more central to XMPP…

Prosody Terminal Utils

Prosody bundles a handful of utilities to help it build a nicer (if still textual) user interface, including:

prosodyctl Supports

In its utility suite the Prosody XMPP Server includes some subcommands for its prosodyctl command. Amongst the less trivial ones (which don’t fit in the main file for these subcommands) are:

All including user interaction via the terminal.

Prosody SASL Authenticators

Prosody’s SASL backend (organized within its “util” junkbox) includes a handful of authentication methods:

Prosody C Code

Looking at the supporting utils implemented in C, there’s: