Re: Obvious (?) question about interoperability

[nospam-seesignature@xxxxxxxxxx]

On Wed, 1 Apr 1998, Hal Finney wrote:

> > can use ordinary libc calls (fread, fwrite).  Otherwise I need a
> > normalization pass.  I think implementations SHOULD store anything except
> > compressed, literal, or conventionally encrypted packets in normalized
> > form.  Further, the conventionally encrypted packets should not divide the
> > first 10 bytes (actually 11) of the stream.
> 
> By "normalization" do you mean the problem of assembling packets which use
> partial-length fields?  If so, it was for this reason that I recommended
> adopting a four-byte length field option.  I understand this will be
> discussed at the OpenPGP meeting tomorrow.

Exactly, but it doesn't matter the length field.  As long as you can have
a valid message of <0xe0>t<0xe0>h<0xe0>i<0x01>s form, evil things happen
to the code.

It is easy to detect, since if the first partial length field is for <4096
I can call a normalizer which would collapse the above to <0x04>this.

> > I would make all of these MUSTs, but embedded applications may have
> > reasons not to, and I can normalize the stream myself.
> 
> What exactly would you make MUSTs?  Do you mean you would disallow the
> partial-length packets?

Disallow the partial length packets EXCEPT in literal, compressed, and
symmetrically encrypted packets (though if any of the other could exceed
the 8192+192 byte limit I would allow this; I could contrive an exaple but
I don't know if one would exist in the real world since such things might
break other static limits - huge MPIs, excessive hashed subpackets, etc.).

Do not allow the partial length packet to subdivide a literal header or
the first 11 bytes of an encrypted packet (I would say 10 bytes, but this
way I can be sure of one byte to decipher which makes the decryption loop
much nicer, and the partial length would be 16 if it met this).

> > Since I introduced the term normalized, I should define it -
> >
> > packet :- oldCTB-len, datachunk... | newCTB len-stream
> >
> > len-stream :- nontermlen, datachunk, len-stream | termlen, datachunk
> >
> > The len-stream is normal if it does not contain any nontermlen entries
> > indicating the following datachunk is less than 4096 bytes.
> 
> I don't understand the significance of this definition.  What is it about
> normal vs non-normal streams which requires them to be treated differently?
> I could see a distinction between packets which have the length at the
> beginning vs packets whose length is not known until you have read the
> whole thing.  But I don't understand why the issue is whether the
> partial length packets are less than 4096 bytes.

For things like [PS]KESKs, signature packets, key material, they are
normally all under 4096 bytes.  They SHOULD all be under the 8192+ size of
the largest terminal packet length.  This allows express handling:

Normal stream:

get packet length of L
/* you can allocate an L byte buffer */
fread(buf,1,L,fp);
/* process buf */

Unnormal stream:

/* make buf *big*, or do a lot of reallocs */
bufp = buf;
get length packet L
while( partial )
	fread(bufp,1,L,fp);
	bufp += L;
	get length packet L
fread(bufp,1,L,fp);
/* process buf */

Or:

Len = pgpfread(buf,1,BUFMAX,fp); /* containing the above */

My 5.X implementation handles things by reading the whole thing in (if
there is a sane upper limit on the size), and then processing by moving a
buffer pointer. 

If you do a series of pgpfread(..) for every subpacket or atomic data
type, it changes things (and I think this is how PGP5.0 does it), But one
problem with this is that you need state information as to how much data
is left before the next expected packet-length, i.e.

Sub: I want 10 bytes
pgpfread: I have 5 bytes buffered, send them, read next packet length, oh,
it is 0xef, malloc 2147483648 bytes :), read them, return 5 more,
remembering that we have 2147483643 left.

Or:
pgpfread: 0xef, read BUFMAX (4096) bytes, and note there are 2**31-4096
bytes left until the next ctb, send the 5 and note there are 4091 bytes
left in the buffer.  pgpfread now has lots of states and if tests to
determine how to respond.  This is ugly.  Really ugly.

I need to do this anyway for indeterminate length types (SymEncPkt,
Compressed, Literal), but having to handle it withing subfields within the
header is painful.  But I can then inline the state info with the
read-decrypt (see my implementation).

--- reply to tzeruch - at - ceddec - dot - com ---