You were on the right track with the Default Programs web page - in fact, it's my reference for this post.
The following adapts their example:
First, you need a ProgID
in HKLM\SOFTWARE\Classes
that dictates how to handle any input given to it (yours may already exist):
HKLM\SOFTWARE\Classes
MyApp.ProtocolHandler //this is the ProgID, subkeys are its properties
(Default) = My Protocol //name of any type passed to this
DefaultIcon
(Default) = %ProgramFiles%\MyApp\MyApp.exe, 0 //for example
shell
open
command
(Default) = %ProgramFiles%\MyApp\MyApp.exe %1 //for example
Then fill the registry with DefaultProgram info inside a Capabilities
key:
HKLM\SOFTWARE\MyApp
Capabilities
ApplicationDescription
URLAssociations
myprotocol = MyApp.ProtocolHandler //Associated with your ProgID
Finally, register your application's capabilities with DefaultPrograms:
HKLM\SOFTWARE
RegisteredApplications
MyApplication = HKLM\SOFTWARE\MyApp\Capabilities
Now all "myprotocol:" links should trigger %ProgramFiles%\MyApp\MyApp.exe %1
.