Re: Dealing with large collections [Re: URI constraints]

[Greg Stein <gstein@xxxxxxxxxx>]

On Wed, Oct 13, 2004 at 08:19:47PM -0400, Robert Sayre wrote:
> Greg Stein wrote:
> >I would not recommend either of those approaches, though I'll certainly
> >give you the "creative" tag :-)  My thinking:
> >
> >* it isn't a range because you're asking for specific ordering. if it
> >  wasn't for that, it would be darned close. the ordering request is
> >  effectively a "content negotation" much like asking for english versus
> >  french.
> 
> I should have been clear that I wasn't trying to match your search 
> grammar's capabilities (though I think the asc/desc is unnecessary). I

Yah, I'm not sure either (it was Ezra's request, iirc, and it didn't seem
"too far out" to go ahead and include it). Conceivably, the app could
rearrange the posts as needed.

> was only trying to match the capabilities of  getRecentPosts() in the 
> bloggerAPI and the metaWeblogAPI. This assumes ordering by atom:updated:

Ah. Yah, that would work.

>...
> If you get a request for the full resource, you can say "202 your export 
> file is here in 15 minutes" or something. Or just give it a 500.

507 (Insufficient Storage) is probably better than 500, but yah.

> >I still like the idea of a teeny search grammar. Each of the parts of the
> >above example would just be an XML element in a query.
> 
> "Want something more advanced? Ah, we have SEARCH and DASL for you."
> 
> My thinking here is that this is the established, already-done way to 
> provide robust, extensible querying. I know that

Well, I'd say that SEARCH has more background for querying (Microsoft
implemented it long ago for Exchange, and catacomb implements it for
Apache/mod_dav). Custom expect tokens, new range specifiers, etc have
almost zero precedent, as far as I know.

> SEARCH/PROPFIND/REPORT/etc don't seem like a big deal to you (or me), 
> but there are *tons* of apps that won't be supporting them for the 
> foreseeable future.

I still find this a bit strange. An application has to insert a custom
header, yet it can't alter the HTTP method? Either way, app changes are
required.

The largest problem is typically proxies, but I'd point out that (for
Blogger at least) I would think most Atom requests will arrive via
SSL/TLS, which also implies a proxy won't be bothered with the method.
(this has been my experience with Subversion and DAV in general)

> I think we should expect that many apps will be 
> taking their existing metaWeblog/BloggerAPI code and adapting. Why not 
> provide a rfc2616-defined way of accomplishing that stuff?

That the examples that I've provided *are* defined by RFC 2616. That spec
Additional HTTP methods are simply one extension mechanism; you're talking
about using other extension mechanisms. Both are compliant with RFC 2616.

In any case... the stake-in-the-ground defines the bits we need to
marshal. It seems we're just proposing a couple different ways to marshal.
All seem valid, so I'm not sure what criteria should be used for
selection. This is probably where "consensus" comes in :-), and we haven't
had much discussion on this yet. Looks like some wiki writing is needed :-)

I think it is also valid to ask Atom client authors whether they need all
of those inputs. Does sort order matter? Are date ranges important? Or is
fetching of N posts at a time sufficient?

Cheers,
-g