Re: Dealing with large collections

[Greg Stein <gstein@xxxxxxxxxx>]

On Wed, Oct 13, 2004 at 09:16:56PM -0400, Robert Sayre wrote:
> Greg Stein wrote:
> >Well, I'd say that SEARCH has more background for querying (Microsoft
> >implemented it long ago for Exchange, and catacomb implements it for
> >Apache/mod_dav). Custom expect tokens, new range specifiers, etc have
> >almost zero precedent, as far as I know.
> 
> Exchange2003 uses a custom range header. I also note this use case seems 
> to be the precise reason that Range is extensible ("chapters of a book" 
> is the usual future case).

And Exchange also uses SEARCH :-)  In fact, the custom range header is
used when searching...

hehe... (sorry for being obstinate)

I don't know that it matters which has precedent, really. All are equally
valid, which is why they're all listed in the new Pace I wrote.

> >I still find this a bit strange. An application has to insert a custom
> >header, yet it can't alter the HTTP method? Either way, app changes are
> >required.
> 
> If you are writing a Python app with urllib2

Oh geez, man. Stay away from that thing. Go straight with httplib. That
damned urllib2 module doesn't even persist connections!

> (yes, I have read your PEP),

Shoot. I wish I had time to work on stuff like that. But that PEP was
really just about formalizing specific extensions to HTTP. It merely
builds on the functionality that I coded up in httplib.

> a J2ME app, a Flash app, etc, you can insert a custom header, but 
> you can't change the method. I'm sure we can find more broken libraries 
> if we want.

Yah. I remember this go-round last time we talked about custom methods. I
also recall having to deal with it when WebDAV first hit the streets. A
lot of stuff didn't deal well with it. Today? All fine. Patience is
needed; I knew DAV would eventually get there, so the transitional bumps
didn't bother me.

People want Atom today, and that definitely poses problems. My preference
would be to define a protocol for the long term using new HTTP methods
where appropriate, new headers to modify existing methods' behaviors when
those methods are close to the right behavior, and/or to define new header
elements (like a new range-specifier) when we're *really* close in
semantics.  THEN, we have a "down-level" alternative for clients that need
to operate in a weak environment.

> The lesson from bloglines/del.icio.us etc is that you have to let people 
> ease into it, IMO. "Get started with standard stuff, we have more 
> advanced things once you're hooked". Think of a beginning PHP developer 
> who just wants to write a quick two liner. I think someone who has 
> written less than 500 lines of code in their lifetime should be able to 
> communicate a bit with the protocol.

I'll note that PHP's HTTP_Request class supports arbitrary methods :-)

The point about enabling beginner programmers could be valid. I just look
at the medium term and think, "heck. there will be a library provided to
make it easier for them." The fact that WebDAV has some more complicated
goo for its property handling is pretty well-covered-up by DAV libraries.
A quick little "import davlib" and away I go.

Take a look at Mark's feedparser module. I have absolutely no interest in
parsing the XML for a feed. I'll just use his module. Done. Very easy. I
believe the same thing will happen for the Atom protocol, so a little bit
of complexity in order to get the model and semantics right is important
to me.

I'm happy to agree to disagree on target audience and simplicity tho. :-)

>...
> >I think it is also valid to ask Atom client authors whether they need all
> >of those inputs. Does sort order matter? Are date ranges important? Or is
> >fetching of N posts at a time sufficient?
> 
> Agreed. And where are these client authors you speak of...?

Beats me. I'd hope they're here.

Cheers,
-g