I just tested this setup. I turned my umami back on, and modified my Caddyfile like I showed you. That last block I had to change to
handle * { abort }
With that, it works as I expected. Tracking works, but my dashboard is only accessible locally from my home network. Outside it, connections are dropped.
@danjones000 Thank you very much, I really appreciate it! I already thought about this a bit and ended up implementing almost exactly what you described (including using caddy) so that's a nice sanity check.
Also, thanks for pointing out the renaming of the routes, I'll go and do that, too.