notes on mailing binary objects

[Robert Ullmann <Ariel@xxxxxxxxxxxxxxx>]

Hi,

There is a common misconception being repeated here: that 8 bit text
is the same thing as binary, and now you can mail anything. This is
simply not correct.

Firstly, text still consists of lines, of some limited length,
terminated by CRLF; the set of symbols available is not 256 in any
sequence.

Much more important is the reliability of the data transfer.

Prime has several years of experience with mailing binary objects
around our network. The following is not theory or speculation, but
is based on hard experience.

Consider the error rate on the net. Specifically, consider the rate
at which /undetected/ errors occur.

If we get one undetected error in 1 million bytes transferred, i.e. a
BER of 10^-7 (using "BER" loosely), this is acceptable for text
messages, but utterly unacceptable for a binary object.

1 million bytes is maybe 500 text messages, our hypothetical 1 error
will cause a typographical error in /one/ of those messages; it will
likely go un-noticed.

Binary messages tend to be larger. The favorite object in binary mail
is an executable, with binary messages averaging closer to 1 MB per
message.

Our hypothetical 1 error in 1 MB is now an error in /every/ message.

Suppose the message was an executable: the result is, at best, a core
dump when run; more likely a subtle failure that might happen a long
time later. Even if you postulate a different actual error rate, it
is still true that binary is 5-6 orders of magnitude more sensitive
to errors in transmission.

The way to move a binary object is for the sender to compute a
checksum (CRC), compress the data, and encode it for transmission.
The receiver decodes, de-compresses, and verifies the size and
checksum.

The object transmitted is normally significantly smaller than the
original binary object, instead of the typical 4/3 expansion from
(e.g.) uuencode.

No changes to MTA's are required: the data is the business of the
MUA's and the users, not the MTA's. Which is as it should be. It
also provides an end-to-end check on the data integrity, which is
crucial to the transmission of binary data.

As I said, Prime has a lot of operational experience with mailing
binaries. The real problem is that it is much too popular; people
mail any object at any time, often regardless of cost ... <grin?>

I will be sending out a draft of a good algorithm for this (the
LZJU90 I referred to previously) sometime in the next week or so.

Best Regards,
Robert Ullmann
+1 508 620 2800 x1736
Ariel@Relay.Prime.COM