Re: UID SEARCH responses

[Mark Crispin <mrc@xxxxxxxxxxxxxxxxxx>]

On Sun, 12 Aug 2007, Pawel Salek wrote:
> I wonder whether it is relevant or not but the copy of RFC3501 that I use [1] 
> does not mention UIDs on page 61... Can you please be more specific?

Page 61 of RFC 3501 is the latter part of section 6.4.8.  That is the 
section that describes the UID command!

> > Unlike message numbers, the UID sets
> > 	38782:41424
> > and
> > 	38782,38789,40004,40301,40421,40814,41424
> > are exactly equivalent if no other UIDs exist within that range.
> "...if no other UIDs exists...". The client may not know whether they do or 
> do not exist.

That doesn't matter.  The server knows, and that exact sequence will 
refer to the messages in question, and only the messages in question.

> The information conveyed to client is  different in these two cases. I see 
> that the basically only thing the client can do with the response generated 
> by your implementation is to send the sequence-set back to the server.

That is what RFC 4731 says as being the purpose of those ESEARCH results.

> It cannot use it to eg. purge local cache for message with UID 40000 
> because it may exist.

Hence there are two other mechanisms to get the UID map which will 
accomplish that purpose

> I cannot help the feeling you are way too dramatic here. I have no idea how 
> IETF deals with imprecise specifications but my impression is it is not the 
> first time there is some dispute how to interpret RFCs.

The IETF generally does not make incompatible changes to a published RFC 
(I-Ds are a different matter) that will have the effect of rendering 
previously compliant implementations non-compliant.  There have been 
lawsuits, and threats of lawsuits, over such matters in the past.

> The usefulness of the 
> response generated by current version of UW-IMAP is strongly limited and 
> considered by me, a client implementer, a bug.

That is your opinion.  I disagree.  I see no point to having the 
complexity of range syntax in returned UID results if it is limited to use 
for those cases where there are three or more monotonically consecutive 
matching UIDs.  I will vigorously fight any ex post facto incompatible 
change to the published semantics of ESEARCH.

> Please observe it does not say "usable ONLY in other IMAP commands. Users 
> must be cautious that the UID sequence sets may be collapsed by the server if 
> separated only by non-existing UIDs". I do not think your argument is valid.

The absence of text in an RFC can not be construed to indicate that the 
RFC takes the opposite position.

The absence of text in an RFC indicates just one thing: that the RFC is 
silent on that matter.

However (and this is important!!) if an RFC does not forbid something that 
is syntactically and semantically valid in the RFC, then that something is 
permitted EVEN IF the author of the RFC thinks that it is bad or wrong or 
stupid behavior.

It is true that RFCs can not forbid every possible bad or wrong or stupid 
behavior.  But if the RFC does not prohibit the behavior in question, and 
that behavior can be reasonably inferred from the specification, then the 
only way to deal with a conflict of interpretation is either to convince 
the other side of the rightness of ones interpretation, or to accept that 
both interpretations are valid.

I have encountered that problem innumerable times.  The majority of IMAP 
client implementations things which I consider to bad or wrong or stupid. 
But, because RFC 3501 (and its predecessors) did not specifically forbid 
the behavior in question, nothing can be done.

In this case, I consider my interpretation to be reasonably inferred from 
the specification AND reasonable in the overall context of IMAP.

> >  . my interpretation returns a valid sequence-set usable in other IMAP
> >     commands
> But your interpretation cannot be used for the same client-side tasks as 
> ordinary UID SEARCH response can.

You can use ordinary UID SEARCH if you need a UID map, and use ESEARCH to 
do traffic-efficient searches once you have the map.

> >  . your interpretation prevents optimization unless there are at least
> >     three contiguous extant UIDs matching the result; without such the
> >     ESEARCH response is larger than the SEARCH result.
> And that's quite common case, I think.

That assumes that UIDs are assigned monotonically by the server, and that 
messages are rarely expunged.

In fact, the implicit requirement in your interpretation that UIDs be 
assigned monotonically by the server is the final nail in the coffin. 
IMAP has NO such requirement.

This is on top of earlier nails, such as the assumption that sequence-set 
means one thing in RFC 3501 and something completely different in RFC 
4731.  Nothing confuses matters more than to have protocol elements that 
change their meaning depending upon context or defining document.  People 
criticize IMAP (justly, in my opinion) for being weighed down with a mass 
of complexity.

> Why not try fixing implementation in question and add a clarification in RFC 
> 4731 errata?

That only works if the implementor agrees that it is a bug in the 
implementation.  He does not.

He thinks that including ranges in ESEARCH is pointless otherwise.

He objects strongly to the idea of IMAP having two types of UID ranges 
with a subtle semantic difference between the two, not to mention adding a 
new hidden requirement to IMAP that UIDs be assigned monotonically.  IMAP 
already has entirely too many subtle semantic differences and hidden 
requirements; this is not a case where "more is better".

> I do not see any client 
> author being happy with UID ESEARCH responses generated by current UW-IMAP - 
> what could it be used for?

It returns shorter search results if you do not need a UID/msgno map. 
You only need to get a UID/msgno map once in a session; after that 
additions to the map are a simple operation and removals from the map are 
easily calculated.  What's more, two other ways exist in IMAP to get the 
UID/msgno map.

Searches, on the other hand, are done many times in a session.  Why should 
those results be weighed down with extra traffic in the assumption that
 (1) the client wants to get the map
 (2) the client does not want to use the two ways which exist in the base
     IMAP specification
 (3) UIDs are assigned monotonically increasing.

> I will be happy, too, with only adding a clarification to RFC 4731 that the 
> UID ESEARCH responses may include non-existing UIDs for the benefit of 
> compression. Such interpetation makes the extension less useful for me but is 
> not a deal-breaker.

Either that, or let it be.

I understand that you would like to have a UID map operation that 
compresses monotonically increasing UIDs into ranges, but does not 
compress UID ranges that are not monotonically increasing.  However, some 
servers do not assign monotonically increasing UIDs; and the operation is 
pointless on those servers compared to the existing UID map operations.

Furthermore, we need to recognize that IMAP's lock into a 32-bit world 
will not be forever.  We are already encountering implementations that 
incorrectly treat IMAP numbers as being signed (and thus limited to 
2GB-1).  The first domino that is likely to fall is the QUOTA extension, 
followed by the base specifications's limit of message size to 4GB-1 
octets.

32-bit UIDs are also in trouble.  Although 4GB messages in a mailbox seems 
unplausible, lots of people have expressed unhappiness with UIDs not being 
globally unique and that creates pressure on a 32-bit space.  Should the 
floodgates of 64-bit be opened, globally unique UIDs will rear their head 
and pretty much preclude monotonically increasing UIDs in a single 
mailbox.

I don't think that any IMAP extension should be designed with implicit 
assumptions that are not in the base specification, but are doomed.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.