Mark Stahl wrote:
One issue that we've encountered with tombtones is that most services don't
keep them forever. (Otherwise, the list of tombstones only ever grows.)
Typically, the limitation on the number of tombstones reflects a limitation in the amount of storage available in a backend service. If tombstone lifetimes are infinite, the storage required for an atom pub collection can grow to an ever increasing size, even if the number of elements allowed in that collection is bounded. Most services I talk about doing AtomPub purge tombstones after some time (for example, 30 days.) So I'm faced with the problem today. Advertising the tombstone window would solve a lot of problems.
This leads to problems with clients that periodically poll for changes -
they don't know if the set of tombstones is complete since the last time
they polled.
I'd suggest adding an element at:deleted-since, so the server can let the
client know the tomebstone window in effect.
To solve this problem, I implemented a simple feed paging extension. The first page of the collection feed (at the collection URI) contains a "feed-changes" link. The feed-changes link points to a feed that is only guaranteed to contain all the information (e.g. entries and tombstones) that was changed since the first page of the collection feed was fetched. The clients can use this to answer the question "what are all the changes that occurred since the last time I synced with the server?"
The first page of the "feed-changes" feed contains another "feed-changes" link that works in an analogous fashion. Clients can keep in sync with the collection by chasing the "feed-changes" links. The "feed-changes" feeds can also use RFC 5005 paging; to get a complete sync, one has to page through all the pages (using "next") before chasing the "feed-changes" link from the first page.
This addresses the large feed document issue, but is still premised on the idea that you store all tombstones indefinitely, so it doesn't solve the problem I'm attempting to solve.
My suggestion is to finish up the at:deleted-entry mechanism that is as simple as possible, and then define a separate AtomPub synchronization protocol on top of it, based on the "feed-changes" mechanism I just described, where the at:deleted-entry mechanism would be a required element of the "feed-changes" feeds. What do you think?
I'm willing to separate the problem of a sync spec from tombstones, so I guess it depends on whether you consider it important that the tombstone spec have the capability to indicate that the set of tombstones provided may not be complete. I tend towards thinking that it's important enough to be included in the main spec, but YMMV, and I'd be happy to have at:deleted-entry today so we can build on it.
Another useful piece of information to the client trying to sync is a hint
as to the typical tomebstone lifetime. But I'm not sure if it belongs in
this spec. (Perhaps it's more appropriate in a service document?)
It is a good idea to keep this information in the feed, because this is useful in Atom feeds that are not AtomPub feeds.
The reason I hesitate to mention this is it seems to be heading more and more towards a sync spec. And I'm not trying to solve the whole sync problem with this spec, but create tools that will be used to solve sync.
Regards,
Brian