Getting ejabberd HTTP file uploads working

[dasyatid1]

I have a tiny Fedora server which I use for some little services, including an ejabberd instance for chat. (This used to be more useful before broad XMPP adoption started drifting away, but still.) The management of it is still pretty ad-hoc after all this time.

I decided I finally wanted to get HTTP file upload tokens working in ejabberd. I'm also using acme-tiny for obtaining TLS certificates—ejabberd has some builtin ACME support nowadays too, but I'd rather keep my ACME certificates controlled in one place and not take on the complexity of nginx→ejabberd proxying for port 80.

This turned out to be a bit more of a saga than I expected.

ejabberd configuration

First I made sure the ejabberd_http entry in the listen list included a request_handlers entry pointing to mod_http_upload…

listen:
  # ...
  - 
    port: 5243
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
      # ...
      "/stash": mod_http_upload

… and adjusted the mod_http_upload configuration itself to match:

modules:
  # ...
  mod_http_upload:
    host: "up.jabber.s.@HOST@"
    docroot: "@HOME@/upload/@HOST@"
    put_url: "https://up.jabber.s.@HOST@:5243/stash/"
    thumbnail: false
    max_size: 104857600

The default configuration prefixes the vhost with “upload”, but I wanted something more XMPP-specific per above.

Then I added the private keys and certificates:

certfiles:
  # ...
  - "/etc/pki/tls/private/up_jabber_s_dasyatidae_com.key"
  - "/var/lib/acme/certs/up_jabber_s_dasyatidae_com.crt"

SELinux adjustment

Unfortunately, it turns out ejabberd can't access the acme-tiny output, spitting out a log entry of “Path /var/lib/acme/certs/up_jabber_s_dasyatidae_com.crt is empty, please make sure ejabberd has sufficient rights to read it” instead. Why? Well, looking at the audit log:

type=AVC msg=audit([redacted]): avc: denied { getattr } for pid=[redacted] comm="10_dirty_io_sch" path="/var/lib/acme/certs/up_jabber_s_dasyatidae_com.crt" dev="[redacted]" ino=[redacted] scontext=system_u:system_r:ejabberd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0

(“dirty_io_sch”, by the way, likely refers to Erlang's ‘dirty schedulers’ for handling long-running native code: see the docs for erl_nif or for the +SDio emulator flag if you want to learn about those.)

Ah, right. So the acme-tiny package doesn't define its own SELinux enforcement. The certificates currently in /etc/pki/tls/certs are of type cert_t, but the ones placed into /var/lib/acme/certs are of the base type var_lib_t. How do we handle this?

One option would be to adjust the file contexts for /var/lib/acme/certs. At a glance, it would seem to make sense for them to be cert_t. Something like that seems more sensible in the long run, but it's harder to analyze in the short run, both regarding whether acme-tiny will run into problems accessing those files as is (though I could manually analyze this by tracing through the systemd unit files and such) and whether it might conflict with later updates to the package.

(Update 2021-09-22: it looks like the Pagure page for acme-tiny agrees that adjusting the file contexts should work, so now I do intend to switch to that. At the time I needed to get the functionality up quickly with minimal risk of breaking acme-tiny.)

I do keep SELinux in enforcing mode on this server, but based on the way Fedora configures things, I treat it as a hardening layer, not as a primary access control layer. So I'm okay with opening a somewhat wider hole in the policy than is strictly necessary until and unless I find a better longer-term answer for acme-tiny. And as it happened, I already had an ejabberd_local SELinux module for handling some other accesses that ejabberd's optional modules might need which weren't covered by the default policy. So let's just punch a larger hole for now…

module ejabberd_local 1.1;
require {
        type ejabberd_t;
        type var_lib_t; # for acme-tiny, sigh
        class file { getattr open read };
        # ...
}

# ...
allow ejabberd_t var_lib_t:file { getattr open read };

One installation later:

# I should really have a script for this...
# Wasn't there a Makefile somewhere that was supposed to help?
% checkmodule -m -o ejabberd_local.mod ejabberd_local.te
% semodule_package -o ejabberd_local.pp -m ejabberd_local.mod
% semodule -i ejabberd_local.pp

ejabberd can now read the certificates, and accessing the appropriate port from my browser results in valid HTTPS.

ejabberd ACL debugging

Unfortunately, it seems like I still can't upload any files as a local XMPP user. Gajim complains of “access denied by service policy” and the XML console indeed shows an appropriate forbidden element. The default is supposed to be to allow all local users, so what gives?

Maybe an explicit “access: local” in the mod_http_upload configuration block will help? Nope.

At this point I decided to go look at the ejabberd code. (At the time I looked elsewhere for unrelated reasons, but I confirmed later with the source from Fedora 34.) Omitting the output:

% dnf download --source ejabberd
% rpm2cpio ejabberd-20.07-3.fc34.src.rpm|cpio -i
% tar -zxvf ejabberd-20.07.tar.gz

In src/mod_http_upload.erl, process_slot_request/6 calls acl:match_rule/3, and then src/acl.erl shows the guts of ACL processing. Setting the log level to 5 (debug) reveals a bit more about the internal messages being passed around. So then let's see what this looks like from the Erlang shell:

% sudo -u ejabberd ejabberdctl debug
... (skipping the warning and also suppressing the input numbering below) ...
Erlang/OTP 23 [erts-11.2.2.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [hipe]

Eshell V11.2.2.5  (abort with ^G)
(ejabberd@localhost)> acl:match_rule(<<"dasyatidae.com">>, local, {jid, <<"rtt">>, <<"dasyatidae.com">>, <<"some-resource">>, <<"rtt">>, <<"dasyatidae.com">>, <<"some-resource">>}).
deny

Whoops. Why? As it turns out, it looks like the local access rule only works in global context:

(ejabberd@localhost)> acl:match_rule(global, local, {jid, <<"rtt">>, <<"dasyatidae.com">>, <<"some-resource">>, <<"rtt">>, <<"dasyatidae.com">>, <<"some-resource">>}).
allow

From the mod_http_upload source, it looks like it's calling this in a vhost-specific context. read_acl in the acl module seems to be the key indirection here, which itself is just an ETS lookup. And indeed:

(ejabberd@localhost)> ets:lookup(acl, {local, global}).
[{{local,global},
  [{user_regexp,{re_pattern,0,1,0,
                            <<69,82,67,80,71,0,0,0,0,8,0,0,1,128,0,0,255,255,...>>}}]}]
(ejabberd@localhost)> ets:lookup(acl, {local, <<"dasyatidae.com">>}).
[]

This seems like it might be a bug somewhere; surely the ACLs are supposed to fall back to the globally defined ones if a vhost-specific one isn't available? In any case, let's try working around it for now by defining vhost-specific ACLs:

host_config:
  "dasyatidae.com":
    # ...
    acl:
      local2:
        user_regexp:
          "": "dasyatidae.com"
# ...
access_rules:
  # ...
  ## Work around weirdness with not checking 'local' ACL globally in some contexts?
  local2:
    - allow: local2

Really this seems closer to what's wanted anyway—a local user at a vhost should have access to its vhost's instances of file uploads and other auxiliary stuff, but not necessarily other vhosts'… from a brief skim through acl.erl it doesn't seem like that's ‘naturally’ supported, but I could easily be wrong.

It works!

And with that, it works. Pushing the attach-file button in Gajim and feeding it an image successfully executes the upload and spits out an ‘aesgcm’ URI which… appears to be based on an HTTPS URI but including an encryption key. A brief Web search reveals XEP-0454, which tries to integrate OMEMO with HTTP File Upload without revealing file content plaintext to the server. The file is being encrypted with a transient key before upload, and then the transient key is being sent over the OMEMO secure channel along with the file reference. Obligatory objection here to jamming crufties into the URI scheme, but Conversations on my handset also retrieves the image and displays it inline, so it seems like at least some interop works. Also, that it still leaks the filename gives me echos of PGP leaking subject lines… sigh.

Future notes to self:

• Is the ACL weirdness a bug in ejabberd, or is my configuration borked? This ejabberd instance has been live for quite a long time, and I wouldn't be surprised if there were some cruft in the YAML file that's no longer the recommended WTDI.
• Is acme-tiny packaged weirdly or incorrectly in Fedora given that it doesn't integrate cert_t or seemingly have any SELinux support at all? (ls -lZ /usr/sbin/acme_tiny says it's just a bin_t too, so I presume it has no SELinux integration.)
• Where the heck did that script for local policy module integration go? I'm sure I had one before.
• Where do I eventually want to go with this management-wise? The mainstream nowadays feels like it's all containers, which has its own fountains of complexity, fragmentation, and centralization in the actual production lines for them. I can't say I'm actually fond of the LSB/FHS style in organization, though—besides which, traditional multi-user with globally-installed packages doesn't map so well to dominant use patterns for servers now, and anywhere between pure-manual and Ansible-style is too clumsily stateful to feel clean. DJB-style installation sadly doesn't have much adoption either. At some point I want to see if Nix is any better…