So now that we've agreed that we need to put the federation into its own layer, separated from the web front-end, I think the next step is to define exactly what we mean by that, and start building up requirements and suggest possible solutions.
I'll get the ball rolling. First, some definitions (feel free to add/correct):
protocol: A protocol is any agreed upon method of communicating between two interested parties. In this case, I would include any API as a particular kind of "protocol" in the general sense.
client: in any situation where there is a request/response interaction going on between two parties, the party making the request is the "client".
federation layer: The top level protocol that allows diaspora pods, client applications, and other social networks to talk to each other and discover what protocols are attached to a user or the pod itself, independent of any user. We say that the federation layer "links" to other protocols. The thing about which this post is written.
protocol link: A link to a particular protocol or version of an actionable protocol. Given knowledge of the protocol and a generic "link" to an appropriate endpoint (a plain URI), a client of that protocol should be able to do something useful with it.
format protocol: A protocol that defines the structure of some particular content irrespective of the context that content is being used. ATOM, JSON, Activity Streams, Salmon I would consider to be format protocols, because they are not actionable by themselves - only when used in conjunction with another protocol. A particular protocol MAY support more than one content format protocol, but this is optional. A format protocol MAY be layered on top of one or more other format protocols, e.g. Activity Streams via JSON or ATOM.
protocol group: a collection of protocols that should be considered together as a unit. This does not include protocols that have their own mechanism for aggregating other protocols (such as OStatus, or the federation layer protocol itself). This is a possible convenience feature of the federation layer protocol to group and version a collection of protocol links when that collection doesn't have it's own aggregation protocol.
Here are the requirements I can think of. Again, please add/correct as you see fit:
The federation layer MUST be able to reference nearly any type of protocol, even protocols that are not aware of the federation layer. This includes non HTTP-based protocols. For instance, it should allow you to link to a telnet server, or an XMPP server, or an instance of an OpenSim server - you name it. If you can make a URI for it, it should be supported.
The federation layer MUST support multiple versions of any particular protocol or protocol group. That means it must have a means of identifying a protocol uniquely by type and version.
The federation layer MUST NOT expect any particular protocol to be present except itself. For instance, authentication should be outside the scope of the federation layer, and should instead be a linked protocol underneath it.
Linked protocols MAY have dependencies on other linked protocols, but the federation layer MUST NOT be responsible for facilitating these dependencies. Instead, this MUST be facilitated by the client and the protocols involved. For instance, one protocol may require an authorization token, which an authorization protocol can provide. The client must then retrieve the token from one protocol and pass it to another. A client may even pass a protocol link to another protocol.