Documentation
Your Service Is Discoverable. That Doesn't Mean Anyone Can Find It.
One of the nice properties of self-hosted software is that it lives on your own network.
One of the less nice properties is that you somehow have to find it there.
When I added local discovery to Everlock, the original problem looked
almost trivial: advertise the service using mDNS and give it a .local
name. There are mature libraries for this, the protocol is well
established, and operating systems have supported variants of
zero-configuration networking for decades.
The implementation itself wasn't particularly interesting.
What turned out to be interesting was everything around it.
A service can be correctly advertised, correctly discovered by the operating system, and still be completely invisible to the user.
And what "visible" means depends heavily on which operating system happens to be looking.
Starting with mDNS
mDNS solves a fairly simple problem: DNS without a central DNS server.
Instead of asking a configured DNS server where a machine is, devices exchange DNS-like queries over the local network. This gives us familiar names such as:
everlock.local
DNS Service Discovery, usually encountered together with mDNS, goes one step further. A machine can advertise not only its name, but services running on it.
For example:
_http._tcp.local.
_smb._tcp.local.
_ssh._tcp.local.
An Everlock instance can therefore say, effectively:
There is an HTTP service called Everlock on this network, running on this host and this port.
No configuration in the router is necessary, and no user has to know the machine's IP address.
For self-hosted software, that sounds ideal.
There is just one problem.
Discovery and presentation are different things
While testing on macOS I initially expected an advertised HTTP service to simply appear somewhere in Finder.
It doesn't.
macOS can discover _http._tcp through Bonjour perfectly well. You can
verify it yourself:
and resolve a discovered instance with:
But Finder doesn't provide a generic browser for every Bonjour service it knows about.
Advertise SMB and Finder understands that this represents something useful to show under Network.
Advertise an arbitrary HTTP server and Finder generally doesn't care.
That distinction is important:
mDNS / DNS-SD
|
service discovered
|
+------+------+
| |
OS knows it OS presents it
Those are two separate capabilities.
I had initially treated them as one.
Then Windows enters the room
Windows has another established discovery ecosystem: SSDP and UPnP.
SSDP also uses multicast discovery, but its model is quite different. Instead of primarily advertising DNS service records, a device announces a type and points clients toward a device-description document.
A media server, for example, can announce itself as a UPnP MediaServer. Windows Explorer understands that concept and can turn it into something the user can actually see.
That made SSDP useful for Everlock's image service.
Suddenly the problem wasn't:
How do I advertise Everlock on the LAN?
It became:
Which representation of Everlock does each operating system understand?
And those are very different questions.
There isn't really one "local discovery"
Once both approaches are involved, the landscape starts looking more like this:
Everlock
|
+--------------+--------------+
| |
mDNS/DNS-SD SSDP/UPnP
| |
+------+------+ +-----+-----+
| | | |
macOS Linux Windows Devices/TVs
| | |
Bonjour Avahi Explorer etc.
And even this is misleadingly tidy.
Support is not binary.
An operating system may understand the protocol but not expose a particular service in its graphical interface. An application may browse one service type but ignore another. A device may understand a standardized UPnP profile but know nothing about a generic HTTP service.
There are therefore at least three questions to ask:
- Can I advertise the service?
- Can the client discover the advertisement?
- Will anything useful happen for the user when it does?
Most protocol documentation answers the first two.
For a product, the third is usually the interesting one.
Standard protocols don't create a standard experience
This was probably the most surprising part of implementing discovery in Everlock.
Using standards doesn't automatically give you uniform behavior.
mDNS, DNS-SD, SSDP and UPnP are all established technologies. The individual protocols aren't particularly exotic. Yet the experience assembled from them is extremely platform-dependent.
A Samba server advertised using Bonjour makes sense to Finder because Finder already has a concept of network file servers.
A generic HTTP server advertised through exactly the same discovery mechanism doesn't necessarily appear anywhere.
On Windows, presenting something as a UPnP device can produce a much richer result in Explorer, but now the device description, its advertised profile and even things such as its icon become part of the integration.
The protocol is only the vocabulary.
The operating system decides which words it wants to hear.
Why Everlock uses more than one mechanism
It would be tempting to pick mDNS because it is simple and use it everywhere.
That would make the implementation cleaner.
It would not necessarily make Everlock easier to find.
Everlock exposes existing protocols wherever possible: Git looks like Git, mail looks like mail, and web services look like web services. Local discovery follows the same principle.
The goal isn't to invent an "Everlock discovery protocol."
The goal is to provide enough standard information that the software already present on a machine has a chance of understanding what Everlock is offering.
Sometimes that's mDNS.
Sometimes that's DNS-SD with a particular service type.
Sometimes SSDP and a UPnP device description produce the better integration.
And sometimes the operating system simply doesn't have a UI for the thing being advertised.
That's fine too.
The useful mental model
After implementing this, I stopped thinking of local discovery as a feature.
It is better understood as a translation layer between a service and the conventions of the local operating system:
service
|
v
standard protocol
|
v
platform interpretation
|
v
user-visible integration
The first two layers are largely under the application's control.
The last two aren't.
That distinction matters when building self-hosted software. It is very easy to test a multicast packet, see the correct response in a debugging tool and declare discovery finished.
Technically, it might be.
But if the machine on the other side knows about your service and the person sitting in front of it still can't find it, you have only solved half the problem.