I'd like to use my NAS download application to handle magnet links. How can I do that with Firefox?

For mailto: links, you can specify web apps like GMail, same for e.g. IRC. However, for magnet: links, this is not possible, only local applications are supported. I even got an error message by the web application offering to configure this, saying the browser didn't support configuring this protocol.