-john
--- Begin Message ---
- To: opensocial-and-gadgets-spec@xxxxxxxxxxxxxxxx
- Subject: A modest proposal for an OpenSocial RESTful API
- From: John Panzer <jpanzer@xxxxxxxxxx>
- Date: Wed, 27 Feb 2008 00:31:32 -0800
- User-agent: Thunderbird 1.5.0.5 (Macintosh/20060719)
All,
I'd like to propose a starting point for an OpenSocial RESTful API specification. The draft below is the result of a lot of individual discussions, a bunch of research and experiments, and, over the past couple of days, a whole lot of coffee. Having said that, I regard this as a way to spark discussion rather than something being put forward for approval. Please reply to the list with your opinions, reactions, and alternatives.
One suggestion: There are a lot of pieces to this (still under 10 pages though). If you start a response thread, it could be useful to edit the subject line to advertise what you're discussing -- the section name, for example. Otherwise discussions could easily be missed.
Have fun,
John
---
Proposal for an OpenSocial RESTful API (DRAFT)
The primary goal of this proposal is to serve as a common protocol understood by all OpenSocial clients and servers. If this draft is adopted, it would replace the previously proposed GData-based OpenSocial data API.
Overview
This API defines a language- and platform- neutral protocol for OpenSocial clients and container servers to interact outside of gadgets on a web page. As a protocol, it is intended to be reasonably easy to implement in any language and on any platform. It should also be usable across a range of clients, from gadgets operating within a web page to servers communicating to synchronize data about a user.
The protocol operates primarily in terms of resources and operations on them. It is defined on top of the HTTP protocol, and uses the standard HTTP methods (GET, POST, PUT, DELETE, etc.) to retrieve and change server state.
No single data representation is ideal for every client. This protocol defines dual representations for each resource in two widely supported representations, JSON [RFC4627] and Atom [RFC4287][RFC5023], using a set of generic mapping rules. The mapping rules allow a server to write to a single interface rather than implementing the protocol twice.
OpenSocial container servers are free to define additional representations but must support at least the JSON and AtomPub profiles defined in this document.
The protocol defines Activity, People, and AppData resources and collections. Most operations consist of retrieving (GET), updating (PUT), creating (POST or PUT), or destroying (DELETE) these resources. The protocol provides a optional feature to batch multiple requests together in a pipeline to avoid multiple HTTP round trips. Finally, it specifies an optional partial update feature which avoids sending large resources over the wire to update just one field.
Data Representations
Each resource has a dual representation as JSON and Atom (XML). All data must be representable in both formats, but we do not attempt to map from generic XML or Atom to JSON. Instead, we define an internal data model using English and JSON syntax, and then define the mappings between this and Atom/JSON.
Each activity, person, and appdata is represented as a hierarchical tree of elements. Ordering of elements within a parent element may or may not be significant, depending on the context. Mapping consists of converting from the internal hierarchy to a JSON data structure or an Atom entry. Examples of the three primary types of data follow:
A Person Example
application/json representation:
application/atom+xml representation:{
'id' : 'orkut.com:34KJDCSKJN2HHF0DW20394',
'name' : {'unstructured' : 'Jane Doe'}}
<entry xmlns="http://www.w3.org/2005/Atom>Person mapping rules:
<content type="text/xml">
<name>
<unstructured>Jane Doe</unstructured>
<name>
</content>
<title/>
<updated>2003-12-13T18:30:02Z</updated>
<author/>
<id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
atom:entry/content == * Flatten entry/content into top level of JSON representation. In the inverse mapping, all otherwise unknown JSON fields are collected into atom:entry/content.In this example, we have chosen to flatten the "content" element by moving all its child elements to the top level. If there were a conflict, the rules could include a renaming for conflicting fields. This renaming scheme, as well as the choice to flatten "content", is driven by metadata defined above for the "Person" service.
atom:entry/id == id (type:guid) Map ids, using a urn:guid: prefix for the Atom representation
/* atom:entry/title == name.unstructured */ This would map atom:title field to the unstructured name; optional but useful -John Panzer 2/26/08 1:56 PM
atom:entry/updated == updated Map updated field; date/time representation is the same in both.
(In general clients can ask for a full representation of a resource, with all fields and sub-fields, or a partial representation, using query parameters.)
An Activity Example
Activity application/json:
Activity application/atom+xml:{
'id' : 'http://example.org/activities/example.org:87ead8dead6beef/self/af3778',
'title' : { 'type' : 'html',
'value' : '<a href="">some activity</a>'
}
'updated' : '2008-02-20T23:35:37.266Z',
'content' : 'Some details for some activity',
'received' : '2008-02-20T23:35:37.266Z'}
<entry xmlns="http://www.w3.org/2005/Atom>Activity mapping rules:
<category term="status"/>
<id>http://example.org/activities/example.org:87ead8dead6beef/self/af3778</id>
<title type="html"><a href="">some activity</a></title>
<content>Some details for some activity</content>
<updated>2008-02-20T23:35:37.266Z</updated>
<link rel="self" type="application/atom+xml" href="" class="moz-txt-link-rfc2396E" href="http://www.google.com/activity/feeds/.../af3778">"http://www.google.com/activity/feeds/.../af3778"/>
<osa:received>2008-02-20T23:35:37.266Z</osa:received>
</entry>
atom:entry/content == content (type:text) Map entry/content to content field in JSON representation, and vice versa.
atom:entry/summary == summary (type:text) Same for summary.
atom:entry/id == id Map ids (representation is the same)
atom:entry/title == title (type:text) Map titles over; if not plain text, use a structured type/value object to represent data.
atom:entry/updated == updated S.O.P.
osa:received == received Map received field, stripping the namespace. In the JSON to Atom conversion, received always means osa:received.
An AppData Example
The first example is of a collection of key/value pairs for a particular application/user pair:
application/json representation:
application/atom+xml representation:{
'pokes' : 3,
'last_poke' : '2008-02-13T18:30:02Z'
}
<entry xmlns="http://www.w3.org/2005/Atom>AppData mapping rules:
<content type="text/xml">
<pokes>3</poke>
<last_poke>2008-02-13T18:30:02Z</last_poke>
</content>
<title/>
<updated>2003-12-13T18:30:02Z</updated>
<author><url>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
<id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
atom:entry/content == * Flatten entry/content into top level of JSON representation. In the inverse mapping, all JSON fields are collected into atom:entry/content.
An AppData Collection Example
In this example, a client has requested a collection of data that spans multiple users. The result is a collection which is given a special default JSON representation as a mapping from users to their data.
application/json representation:
application/atom+xml representation:{
'orkut.com:34KJDCSKJN2HHF0DW20394' : {'pokes' : 3, 'last_poke' : '2008-02-13T18:30:02Z' },
'orkut.com:58UIDCSIOP233FDKK3HD44' : {'pokes' : 2, 'last_poke' : '2007-12-16T18:30:02Z' }
}
<feed xmlns="http://www.w3.org/2005/Atom>AppData mapping rules:
<id>...</id>
<title>...</title>
<entry>
<content type="text/xml">
<pokes>3</poke>
<last_poke>2008-02-13T18:30:02Z</last_poke>
</content>
<title/>
<updated>2008-02-13T18:30:02Z</updated>
<author><url>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
<id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
<entry>
<content type="text/xml">
<pokes>2</poke>
<last_poke>2007-12-16T18:30:02Z</last_poke>
</content>
<title/>
<updated>2007-12-16T18:30:02Z</updated>
<author><url>uurn:guid:orkut.com:58UIDCSIOP233FDKK3HD44</url></author>
<id>urn:guid:orkut.com:58UIDCSIOP233FDKK3HD44</id>
</entry>
</entry>
atom:feed/entry/author/url (type:index guid) Use the author/url field as the unique key for the JSON collection; on inverse mapping, take keys and store in author/url, using a guid mapping that adds urn:guid: prefix
atom:entry/content == * Flatten entry/content into each record in resulting set.
The default representation of a collection would simply be a JSON list of records ([ {...}, {....}, ...]). The special type "index" means that the collection is instead to be represented as a mapping, using the index field.
Collections
Collections are a useful abstraction for dealing generically with multiple things. They again have both Atom and JSON representations; the Atom representation is simply a feed whose entries are as specified above. The default JSON representation is a JSON list of JSON objects, with the individual objects as specified above.
Activity stream collections are ordered chronologically by default, most recent activities first. Other orderings can be requested for collections using query parameters.
Operations
OpenSocial uses standard HTTP methods: GET to retrieve, PUT to update in place, POST to create new, and DELETE to remove. POST is special; it operates on collections and creates new activities, persons, or app data within those collections, and returns the base URL for the created resource in the Location: header.
Discovery
This proposal specifies URL template rules for discovering collections; the expectation is that all containers can support the same URL paths and that the rules for composing URLs are simple enough to avoid the need for generic discovery mechanisms in the basic protocol. Should this prove impossible, more flexible discovery mechanisms can be added.
Activities
URL patterns:
/activities/{uid}/self -- Collection of activities for given user ({uid}=me is a placeholder for currently logged in user)
/activities/{uid}/friends -- Collection of activities for friends of the given user {uid}
/activities/{uid}/{groupid} -- Collection of activities for people in group {groupid} belonging to given user {uid}
/activities/{uid}/self/{aid} -- Individual activity resource; usually discovered from collection
Modifiers:
?app={aid} -- Restrict the collection to a given app's activities ("me" is a placeholder for the app making the current request)
People
URL patterns:
/people/{uid}/all -- Collection of all people connected to user {uid}
/people/{uid}/friends -- Collection of all friends of user {uid}; subset of all
/people/{uid}/{groupid} -- Collection of all people connected to user {uid} in group {groupid}
/people/{uid}/all/{pid} -- Individual person record.
/people/{uid}/self -- Self Profile record for user {uid}
/groups/{uid}/self -- Collection of groups owned by the user, which always contains 'all' and 'friends' and may contain more. (Details TBD)
App Data
URL patterns:
/appdata/{uid}/self/{aid} -- Individual App Data record for a given user+app, consisting primarily of a bag of key/value pairs.
/appdata/{uid}/friends/{aid} -- Collection of App Data records for friends of {uid}.
Additional Standard Query Parameters
These query parameters may be used with any URL above.
format={format} -- Format desired; one of (atom, json-c); default is json-c.
fields={field+} -- List of fields to include in representation or in the members of a collection.
startPage={startPage} -- Index into a paged collection
count={count} -- Set page size for paged collection
Authentication and Authorization
Requests use OAuth [OAuth.net] for authentication and authorization. The container is the OAuth Service Provider and the client is the OAuth Consumer. Clients can use and pass in a full user-level authorization token, using the full OAuth protocol. Alternatively, if a client has a trust relationship with a container, it can use "two legged" OAuth authorization in which there is no user-based token. (A "two legged" use profile is being defined for within the OAuth working group.)
In each of these cases, the URL and auth[nz] data is signed by the OAuth protocol. Additional security can be obtained with SSL connections.
Clients may supply an OpenSocial-specific OAuth extension parameter, x-oauth-os-uid, to identify the current user (the user who the client is acting on behalf of). The format of this parameter is the urn: GUID of the current user, as supplied by the container to the client in an out of band manner.
Concurrency Control (Optional)
All resources use the HTTP/AtomPub ETag-based optimistic concurrency mechanism. Servers provide ETags for resources when they are retrieved, and clients supply If-Match: headers with the etag when they perform modifying operations. If the resource has been modified between the initial retrieval and the update, the update fails with a conflict HTTP code.
A server which chooses not to support optimistic concurrency should omit ETags on its responses.
Batching (Optional)
There are use cases where clients want to send several operations, in a specified sequence, to a server to execute in turn, but do not want to incur the overhead of multiple HTTP round trips. A specific use case is a gadget which first adds a new friend, then retrieves the 5th page of friends (what it's currently viewing). These are two ad hoc requests from the point of view of the server involved, but for optimization purposes the gadget wants to batch up the requests and responses to minimize latency. It is also possible that the requests and responses would be to different types of data (activities, people, or persistence) and/or to different servers. In the most general case it would be useful to be able to batch any type of request, including requests to third party servers (e.g., "phone home" requests) that need to be executed in sequence with container-local requests.
The semantics of a set of batched requests in this proposal are exactly the same as sending the same requests individually. In particular, the server does not provide any atomicity guarantees. The server is not allowed to reorder or parallelize the requests unless it can do so without client-visible side effects. The simplest server implementation is to loop over the requests, executing each in turn, and appending responses into a batch response to send to the client.
The batch interface is an optional part of OpenSocial. If a server does not support it, a client falls back to using individual requests and responses with no changes in semantics. Depending on the exact situation, it may be more efficient to use individual requests which can be parallelized and cached, or to use batch serial requests; a goal with this API is to make it simple for a client to switch between the two mechanisms and to keep the semantics consistent and straightforward.
To this end, a batch request is represented by an envelope HTTP request sent to a standard proxy resource (/batch_proxy, in this proposal). The client POSTs the document described below to the proxy URL, which returns 200 OK when the contents have been processed (successfully or not) and returns a response sequence document.
Both request and response sequence documents are represented by standard multipart/mixed documents. The request:
POST /batch_proxy HTTP/1.1
Host: container.org
Content-Type: multipart/mixed; boundary=batch-a73hdj3dy3mm347ddjjdf
-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Operation: POST /people/me/all HTTP/1.1
Host: container.org
Content-Type: application/json
...representation of new Person elided...
-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Operation: GET /people/me/friends?startPage=5&count=10&format=json-c
Host: container.org
If-None-Match: "837dyfkdi39df"
The response:
HTTP/1.1 200 OKIn this case, the friend add succeeded and the response to the retrieval of page 5 of the friends list indicates that page 5 didn't change (the friend was added below that page's last entry). If the resource had changed, of course, you'd see a list of the friends in the response.
Content-Type: multipart/mixed; boundary=batch-a73hdj3dy3mm347ddjjdf
-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Status: 201 Created
Location: http://container.org/people/me/all/container.org:5ea3gh838kjk34834
ETag: "993hhhgjgkkd"
-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Status: 304 Not Modified
ETag: "837dyfkdi39df"
X-Batch-Operation: Contains the HTTP request line verbatim with verb, URL path, and HTTP version.
X-Batch-Status: Contains the HTTP status line verbatim with response code and any following text.
Only the top level request can specify overall request related things such as keepalives and content transfer encoding. The top level request response code is:
- 200 if the batch was processed, even partially;
- A 4xx error if the batch document itself was bad and could not be processed (note that a bad document might not be detected before partial processing occurs, so clients cannot rely on the server to do syntax checking). An example of this would be a document that is not of type multipart/mixed.
- A 5xx error if the proxy batch service itself is having severe problems.
Individual responses use the HTTP response codes they would have used had they been executed individually. Requests and responses are matched by position, and the server guarantees that if it sends back a response, the ith response is for the ith request. It does not guarantee that the number of responses matches the number of requests; in particular, network problems might truncate a response document or drop a connection, and in this case clients are left not knowing the status of the remaining responses. If a server needs to indicate that a particular sub-request timed out without any indication of whether it succeeded or failed, it should use "X-Batch-Status: 504 Gateway Timeout" to indicate that it has a known unknown. Clients seeing a 504 sub-response should be advised that the request may have completed before timing out.
Clients may discover whether a server supports batching by performing a GET or HEAD on the well known /batch_proxy URL endpoint. The server response should include an Allow: header which contains POST if the server supports batching. The response should be marked as cacheable so that clients can avoid round trips at startup.
Note that a container may support proxying to foreign servers (via the Host: header); however, servers are not required to support this and clients should not assume the capability.
In general, HTTP fields may be added to either the envelope request or sub-requests. For example, a client may add an Authorization: header to the envelope request (to authorize itself to the proxy service) and also add Authorization: header(s) to different sub-requests (to authorize itself to other services).
References: http://www.snellspace.com/wp/?p=788, http://www.intertwingly.net/wiki/pie/PaceBatch
Partial Updates (Optional)
Partial updates avoid the need to send full representations of data on updates, especially for People and App Data. The primary use case is when a client has retrieved a full representation and subsequently wishes to send a small update to a server. In the most general case, the client my have a series of small updates (e.g., changing tags on a large number of contacts) scattered across many resources. It should also allow for concurrency control. We wish to have a generalized partial update / patching mechanism to handle such cases. As with the batching mechanism, it should be optional on the server side; clients can always fall back to sending full representations back to the server if it does not support the mechanism.
The exact syntax is TBD at this point.
--- End Message ---