Re: MODSEQ in CONDSTORE

[Dave Cridland <dave@xxxxxxxxxxxx>]

On Thu Feb  2 17:23:04 2006, Mark Crispin wrote:
> On Thu, 2 Feb 2006, Dave Cridland wrote:
> > A user setting \Seen (possibly implicitly in a FETCH BODY[...]) 
> > causes the MODSEQ to be bumped for that message. If the 
> > implementation is poor, then this can cause the MODSEQ seen by 
> > other users to be changed, too. That's undesirable, but as a 
> > client implementor, I infinitely prefer that level of 
> > implementation to none at all.
>
> But that's only if (1) the mailbox has per-user flags, (2) the 
> mailbox has MODSEQ.  Right?
Exactly, that's precisely the circumstance I'm describing.


> Personally, I never thought that per-user flags were worth the 
> complexity, even though I was guilty of creating them.  But it was 
> only as a grim necessity for bboards and newsgroups; and even then 
> that was just a single flag.  The idea of using MODSEQ for such a 
> thing staggers me.
It seems quite logical to me. I'm told it's a little tricky to get 
right/efficient with CONDSTORE, mind.


> > Otherwise, you're effectively lowering the width of the UID space, 
> > down to 30, or 22, bits. That's still probably plenty of space, 
> > but there's something philosophically wrong about an extension 
> > lowering the scalability of the protocol it extends.
>
> This argument only applies when you are monotonically allocating 
> UIDs (otherwise it's your implementation that reduces the UID 
> space, not MODSEQ).  And if you are monotonically allocating UIDs, 
> then there is no reason not to monotonically allocate MODSEQ.  The 
> notion of there being over 4 billion updates to one mailbox is 
> simply not credible.
Probably not right now, but the notion that if you allow for 
4,294,967,295 messages in a mailbox in the protocol - and never mind 
that existing servers will collapse long before that - and the 
arrival of each counts as a modification, and there can be further 
modifications, therefore that the maximum number of modifications 
before an overflow event should be higher than 4,294,967,295 seems 
perfectly sensible to me, especially when no "real-world" server has 
to support more than 4,294,967,295, it's only the protocol.


> > > I disagree.  Either MODSEQ reports that something touched the 
> > > message, or it reports that something changed the mesage.  As 
> > > matters stand, it is ambiguous.  A client may care.
> > Well, let me put it this way - a client which actually does care 
> > is broken. A server which claims there to be a change when there 
> > is not is deficient.
>
> [sarcasm] And the IMAP community never, ever, produces broken 
> clients and deficient servers (or deficient clients and broken 
> servers). [/sarcasm]
>
> We should have learned the painful lesson by now.
Sure... But the deficient server in this case causes no damage other 
than a spurious round-trip from a client.

I don't see how a client could assume that a MODSEQ change to a 
message represented a visible change to the FLAGS, but even if it 
could, what good would it do? Surely in finding the MODSEQ, it could 
easily enough find the FLAGS at the same time? And if it did make 
such an assumption somehow, what would it then do which would affect 
other clients?


> > > > Well, I don't think it's much of a problem. Most of the 
> > > > computers I work with these days increasingly use "unsigned 
> > > > long" for 64-bit values - even my laptop is 64-bit, but the vast 
> > > > majority of the 32-bit systems I use seem to support some kind 
> > > > of "unsigned long long" or equivalent.
> > > That hasn't been my experience... :-(
> > Get a better compiler? :-)
>
> Sure, please write it.  And give it away for free, along with free 
> technical support.
To be fair, there's not much point competing with the GCC community, 
who seem to be doing a marvellous job there.


> I have a difficult time tolerating the "my system has 
> such-and-such, and it's your fault if your system does not have it 
> too" attitude.  Either we care about interoperability and 
> implementation costs, or we should forget this IETF nonsense and go 
> home.
Mark, I obviously care about interoperability, as we're right here 
discussing it. I obviously care about implementation costs, because I 
provided an estimate. I'm also already at home, come to that.

I'm simply saying that many systems do, indeed, have 64-bit unsigned 
integer datatypes available, and that it appears to be an increasing 
trend, and I actually find it very hard to believe that there exists 
any current desktop system which does not have 64-bit support in the 
compiler and libraries.

There are indeed some systems that don't, such as Symbian for one, 
but in my considered opinion, this doesn't matter, because it only 
affects client implementations, and client implementations can cope 
perfectly well with using 3 strings, for only a slight overhead.

[Aside: Symbian's compiler is GCC, and does indeed have unsigned long 
long in at least some versions, but the descriptor library has no 
support for it, only limited support for a C++-implemented signed 
64-bit int.]

The implementation cost is therefore very slight indeed, unless the 
client stores every MODSEQ's value and/or performs arithmetic or 
comparisons upon them, and this is unlikely to be beneficial to the 
client, and it is certainly not required for the purposes of faster 
resynchronization, which is the obvious use-case for CONDSTORE on 
limited platforms anyway.

On those platforms where it's possible to build a usable server which 
could overflow a 32-bit MODSEQ, whether by an astonishing throughput 
of STORE commands or a truly impressive amount of exceedingly fast 
disk, I refuse to believe that a compiler with 64-bit support is 
going to be a problem to find.


> > I don't believe clients ever need to handle more than 3 MODSEQ 
> > values at any time, and only need to store 2 permanently. If I 
> > needed to store each message's MODSEQ, I'd probably feel 
> > differently, but I don't think that's ever the case.
>
> Are you certain?  IMHO, the client ability to fetch a message's 
> MODSEQ directly is like placing a loaded pistol with a hair trigger 
> in a room of preschoolers.  MODSEQ's ascending property makes it 
> nice and shiny too.
Well, no, I'm not certain there are no cases where being able to 
treat the 64-bit MODSEQ as a 64-bit number is useful to a client, 
merely that I've not found one useful enough to implement yet. Nor 
have I found the ability to directly find MODSEQ values for messages 
useful, as yet. There's no harm in doing so, of course.

However, I've implemented the client side for the uses of fast 
resynchronization, and I use it exclusively and daily to read my 
mail, and it works perfectly well, dealing with only three MODSEQ 
values per mailbox. Nor can I see anywhere in my code where instead 
of using a true 64-bit integer and comparing, I couldn't just as 
effectively compare strings for equality. I'm very confident that 
adding the conditional STORE itself would not cause me to change that 
assessment.

I have spotted two possible cases for the rare circumstance where a 
difference of one is noted between HIGHESTMODSEQ values which 
represent a potential gain, when also comparing UIDNEXT and EXISTS, 
but it's so rare I've never bothered implementing it, and I strongly 
suspect that, again, this is possible to handle with strings.

Dave.
--
          You see things; and you say "Why?"
  But I dream things that never were; and I say "Why not?"
   - George Bernard Shaw