Reviewing philosophies and assumptions

[KLENSIN@xxxxxxxxxxxxxxx (John C Klensin)]

I've been away for a few days, as some of your have probably noticed
:-) In the interim, several interesting things seem to have happened.
Some of it is beginning to sound as if we are converging.  I don't
think that is the case.  So, in the spirit of Dan's "Basics" proposal,
I would like to suggest a radically different philosophical view, as a
strawman, if nothing else.

I suggest that, in the aggregate, what is proposed below constitutes
keeping SMTP more simple than Dan's suggestion.  I favor "really
simple".  It prevents, as others have been pointing out, the potential
of serious interoperability problems.

Some of the ideas below are stolen shamelessly from various other
messages and private communications.  I don't claim credit for anything
other than drawing this together, perhaps badly.
  ---------------------------

Part I.  Different service, different port.
  I think this is a terrible idea for SMTP, since, IMHO, it will
increase the odds that someone on one host won't be able to send
minimal mail to someone on another host without knowing a lot.  I am
opposed to anything that increases those odds, since having mail as our
minimum mode of communication is vitally important.  If we lose that,
we don't just kill SMTP, we might as well all go back to our fax
machines (apologies to John McCarthy).
  I think this is a wonderful idea for multimedia, multipart, binary,
etc., message-passing.
  Those two beliefs are not contradictory.

  Alternate philosophy: Let's assume that X.400 is coming, that sooner
or later, we will be supporting it, in all of its encapsulated message/
multimedia glory.  If that is the case, the sooner we get started, the
more we are likely to understand and start to fix what is broken.  If
we create a new port, and we designate that port, not "really-extended-
SMTP", but "X.400-over-TCP/IP", we start a move in the right direction.
We also reduce the number of wheels that get re-invented.
  This is not a digression from "extended SMTP", but a powerful
heuristic for focusing the mind.  If one *assumes* an Internet X.400
service, then the question for the SMTP extensions effort changes from
 "how much can we stuff into SMTP?"  into
 "where should be boundary between the two be drawn?" or "given the
X.400 capabilities, how much effort is it worth putting into SMTP?"
  It seems to me that much of the discussion on this list has
addressed, and re-addressed, the first version, despite (what I've
interpreted as) Greg's attempt to set a much more narrow agenda.  I
think that the latter questions would move us back to his agenda.  The
approach of trying to draw a boundary between two known alternatives--a
stretched SMTP/RFC822 and a presumably-relatively-complete X.400--would
do a better job of focussing on Dave's recommendation of figuring out
what is important than lots of folks saying "what I really need is
pictures", "what I really need is sound", "what I really need is
unlimited Han".  Each of those may be true, and valid, and maybe the
rest of us need them too.  But it seems to me that it would be helpful
to look at these options in the context of "if you need that, and you
can get people who want to communicate it with you using it, you have
it over there in X.400.  Can you make the case that it is also
important enough to make SMTP more complex and either to insist that
everyone support it or to risk decreasing interoperability?".

  Note that this principle does not say "no encapsulation" or "no
binary" or whatever.  It just says "here is the alternative, do we need
to complexify SMTP and our expectations of SMTP implementations by
putting this stuff in?".  The answer has to weigh the amount of need
against the amount of complexity needed to achieve the result.

 ------------------------

Part II.  What is a "mail"?
  Others, including Dan, have periodically written down "SMTP" and
stressed "Simple".  I think that is worthwhile, and have repeated it
above.   But I think it may also be useful to stress the "Mail" part.
It is interesting to note that, in the ISO-world, X.400, with all of
its embedding, multiple-everything, potentially non-interoperable
options, etc., isn't called a "mail" protocol.  "MOTIS" is "Message
Oriented...".  Not "mail", but "message".  And not even "message", but
"message oriented".  As a philosophical strawman, think about "mail" as
being that which can be prepared on a cheap typewriter.  That has
been the Internet definition, in practice, for as long as I can
remember (and my memory goes back to the week the MAIL verb went into
FTP :-) ).  This does not take away from the fact that various clever
schemes have been created and used that permit, with the agreement of
the interchanging parties, all sorts of other things to be "keyed into"
those virtual cheap typewriters and decoded at the remote end, and that
several of them have made it into RFCs with a variety of statuses
(other than mandatory Internet Standard).
  Now, the big advantage of "simple mail", as stated above, is that we
have ended up with a surprisingly effective "everyone can interoperate
with everyone else, even across rather stupid gateways" situation.  It
is not optimal in lots of small ways, but it works.  It would be very
unfortunate to "break" that in the process of making it better.  Seems
to me that this makes a strong case for preserving "simple mail" as
"simple mail" and putting, e.g., binary and multimedia extensions,
somewhere else, where they can't break the mail.  Not a bias against
those things--I think they are mostly good ideas and the Internet
should have them--just the kind of extremely conservative engineering
that I think this situation requires.
  Viewed in that light, the main goal of this extensions exercise, as
I understood what Greg laid out, is to improve the cheap typewriter to
give it multilingual, not just ASCII, capability.  It is not to turn
the cheap typewriter into a high performance multimedia workstation.

  And nothing about "mail" requires encapsulating things.  Dave Crocker
pointed out
  > What we have seen in the recent discussion is that touching 8-bit
  > draws you into the tar baby of multiple body parts and different
  > types of message content.
It is certainly true that the "recent discussion" keeps coming back to
this.  It is also bogus.  Touching 8-bit implies touching 8-bit.  The
fact that people see that as an excuse/opportunity to grind other axes
is fine, but it does not follow.

 -------------------

Part III. Whither 8 bit transfer?
  The idea of all of those 7-8 and 8-7 bit conversion agents lying
around really disturbs me.  I think the potential for
non-interoperability is large, as others have pointed out.  It is worth
noting that there are versions of UUDECODE (one of the suggestions)
around that will only "mostly" handle what unrelated implementations of
UUENCODE create.  Interactively negotiated options, a la Telnet, are
pathological over long-latency transfer paths and various other
situations.
  One possibility is to say, more or less as someone suggested
recently, "it is just bandwidth, let's force everything into a seven
bit path and at least avoid all of the uncertainty and negotiation".
That argument is very persuasive.  I think it is also wrong.
  Let's try the other version of the same thing, again at a
philosophical level...
  TCP provides an eight bit path.  Telnet can provide an eight bit
path.  Any SMTP server that is going to accept extended SMTP is going
to need to be altered to accept a new verb or two (we have agreed on
that, haven't we?).  Any SMTP sender that is going to push extended
SMTP is going to need to be altered to send those verbs.
  Strawman: At the transport, receipt, and sending levels, an extended
SMTP server supports 8 bit data paths.  If it does not do that, it
isn't an extended SMTP server, it rejects the verbs(s), and it is held
responsible for 7 bit ASCII only.  What an extended (8 bit) SMTP server
does to transfer 8 bit information into a 7-bit-character-host is up to
it and the host, and none of the Internet's business.
   Relay hosts are not required to support anything they don't support:
if they support extended SMTP, then they are expected to relay extended
SMTP over eight bit paths.  If they don't, they they aren't.  If I'm a
host who can support extended SMTP, but I get my mail via a relay that
doesn't, well, life is hard and maybe I need to look for another relay.
Mail eXchanger hosts are, as always, free to work out whatever mail
transfer arrangements they like with hosts for which they provide mail
exchange service.  In a fashion more like "translate to internal
character set and file system" than like "relay mail", they might
encode into seven bits, or reverse the bits from right to left, or
whatever.  None of the business of the Internet protocol suite.
  The important thing here is that there are no seven bit data paths
over TCP.  That is a limitation imposed by SMTP servers or clients
(traditionally on behalf of their host file systems, but that is not
the issue).  And it is only over TCP and Telnet that SMTP is required
to be supported today.  If someone wants to support an extended SMTP
analogue over a real seven-bit data path, then the Right Thing is for
them to invent SMTP8-7 for transporting mail over that path, not to
distort the Standard extended SMTP, and raise the risk of reduced
interoperability, to accomodate the properties of their non-Standard
transport. 

  Now, one can get rid of much of that by adopting the symmetric
argument, which is "7 bit always, you never need 8 bits".  That is, to
me, second-best.  2x2 matrices are worse.  Much worse.

  Heavy stuff, huh?  But simple...

  Unfortunately, even saying "seven bits only" does not prevent having
to change SMTP...

 ------------------------------

Part IV.  Another look at character sets.
  If we are going to accomodate character sets other than ASCII, there
are, as far as I know, four approaches suggested by various existing
and proposed International Standards.  They are:
  (i) Stick with old versions of ISO 646 and its national variants.  If
we do this, we need to only identify the national variant.  We also
don't need an eight-bit data transfer path.  This approach has largely
been a failure, and is included for completeness only.
  (ii) Use ISO 2022 and the designation sequences against any
registered character set.  The problem with this plan is that it is
very hard to support properly, and "hard to support properly" is nearly
equivalent, IMHO, to "promoting a loss of interoperability".
Fortunately, no one has suggested it lately, so let's ignore it.
  (iii) Make the leap to a "universal", potentially multi-octet,
character set.  Let's call this 10646, with the understanding that
July's 10646 might differ a bit from today's 10646.  The story here
does not change significantly if we were to choose some other candidate
universal character set, and I've probably said enough about the
uncertain state of DIS 10646 itself and won't repeat that here.  Maybe
it is important, maybe not.
  (iv) Use a family of strictly-single-octet character sets.  Let's
call this 8859-n, with some of the same qualifications as apply to
10646 (although the 8859-n family is pretty well populated by Standards
today).

Now, despite the tone of "get it to the SMTP server, what happens
inside the host is not an Internet problem" stated above, let's look at
the end-to-end message process.  10646 is going to be relatively hard
to support, at least without a negotiation protocol that says "send me
only these subsets".  This assumes that "support" implies "actually
display the stuff and permit typing it in", not just "have some ugly
kludge of an escape arrangement that permits the user, somehow, to
determine what was sent".  Now, quick, everyone look around their
offices for 10646-complaint terminal or display devices.  And 10646-
producing keyboards.  And editors, programming languages, and operating
systems that are really comfortable with variable-number-of-byte
character sets.  Some of us have them, most don't and won't for a
*long* time.  If anyone has a UA written in some sort of command-macro
language, think about what it would do it it encountered a control
sequence that is equivalent to "ok, until I tell you otherwise, all
characters are two (or three) bytes long".  If you don't find a lot of
these devices, and "sure, we do that" answers to the systems questions,
think carefully about 10646.
  Also remember that the embedded shifting and designation sequences
are very sensitive.  If they get garbled (note that these are single
character errors), you may end up looking at Thai when you expected
Greek.  An uncomfortable conversion experience, I expect, and a strong
argument for checksums or ECC.  Do those belong in SimpleMTP?  I think
probably not.
  I'm not suggesting that we aren't smart enough to use 10646 and make
it work.  It is just that it is "not simple": either one agrees to
swallow the whole thing, which is hard, are one has to negotiate which
subsets will be accepted, which gives us most of the disadvantages and
none of the advantages of doing something else.

  The idea of SMTP servers taking 10646 (or anything else) and
translating the codes and graphics into nearest-local-approximation is
very frightening.  First of all, to figure out what nearest-local-
approximation is, one has to understanding every code position in the
character set, some of which are not filled in yet.  Second, these
decisions really need to be made as close to the sender as possible.  I
don't read Turkish, not a bit of it.  If someone wants to send me a
message in Turkish, whether or not I can accept the Turkish characters
in 10646 (or 8859-2, for that matter) is going to be the least of my
problems.  The protocols should provide me and my correspondent every
mechanism we might work out among ourselves for dealing with this
problem, but no SMTP server or UA should be expected to make the choice
between bit-stripping to ASCII (yeech), pretending that Latin-2 is
Latin-1, transliterating the Turkish pronounciation of the words into
something English (or IPA)-like, or actually translating the message
into English.  At least it should not make that choice without
per-user, per-message, information.  And, in most cases, either I'll
want to accept the Turkish, write the mail into a file, and try to find
a translator who also has the hardware and software to display the
message, or we are going to get it translated at the originating end,
and send a language I can read (both in character sets and in meaning).
  Note that none of this has anything to do with whether 10646 is sent
over an eight bit channel or is coded somehow into 7 bits.  It has to
do with the graphics, semantic meanings, and locations of the
characters. 

So, let's look at (iv) again, since it is the remaining alternative.
Suppose we make a rule that "extended SMTP" replaces "7 bit ASCII" with
the following principles:
 (a) An extended SMTP server MUST accept ISO8859-1 (Latin-1) as a
character set.  For extended SMTP, ISO8859-1 becomes the lingua franca,
corresponding to ASCII in [unextended] SMTP.  Perfect? No, but a *big*
improvement over the lingua franca of X.400.
 (b) An extended SMTP client MAY announce an intention to send mail
containing any other character set from the 8859-n family.  An extended
SMTP server MAY accept that assertion of intention, in which case, it
assumes the same obligations for accurate and undistorted delivery as
it does when it responds "250" to a RCPT TO.
 (c) An extended SMTP server MAY, alternately, reject anything but
8859-1.  If it does, the extended sender MUST NOT send anything but
8859-1 or its ASCII subset.  Whether it compiles with this principle by
giving up and closing the connection, by sending in some encoding
system agreed upon with the receiver, by transliteration, or by
translation is outside the scope of the protocols.

Now, what does this bizarre notion buy us?  First of all, it gets
almost immediate interoperability, one level out.  ISO8859-1 is a nice,
simple, eight bit (single octet) character set, with no provision for
suddenly turning into a multiple octet monster.  There are lots of
cheap terminals and not-so-cheap operating systems that support it.
Many implementations of SMTP servers would need to be changed only to
accept the relevant verb.
  Accepting the other members of the 8859-n family permits (or can/will
permit) supporting all of the characters that 10646 would provide.  I
would certainly suggest that an extended SMTP server that is relaying,
or that does not know a lot about the receiving UAs and devices it
supports, might want to accept any 8859-n set.  It would certainly be
no worse off than accepting 10646, and wouldn't need to worry about
multibyte characters and switching sequences.  And, by knowing,
clearly, what is coming, UAs could reject if MTAs didn't--I'd expect
"5xx I can't make any sense of that character set" to become a message.
I think that is desirable, if the alternative is total muddle and no
way to cope with it.
  Let's see what the 10646 alternative would be like: "5xx I can accept
this subset, and that subset, but not any subsets that involve more
than three bytes per character..."  Ah, but decision-making is on the
codes, not the strings, right?  So we are going to have a whole
hierarchy of codes for different 10646 components that can't be coped
with?  The easy way to avoid that problem is that everyone, down to the
UAs, is required to support all of 10646.  I think that does not lead
to a lot of working implementations in a hurry.
  But this fancy stuff would not become a requirement.  By keeping
things significantly more simple, and defining a new common character
set, we encourage lots of implementations, done rapidly, and we
preserve, IMHO, the level of interoperability that we now have with
SMTP.

  It does, as a strategy, have some costs.  If one needs to write a
message that contains text in several different languages and
corresponding character sets, some kind of embedding will be needed.
This is a "one message part, one character set" proposal and, if
embedded/compound messages are pushed off onto X.400, it becomes a "one
mail message, one character set" proposal.  10646 does not have that
problem: one can arbitrarily switch character sets in and out
mid-message if the designation sequences are permitted, as Dan
suggests.  I'm not convinced that facility is important enough to
justify the added complexity and the fears of interoperability
problems.  One can always do that stuff in X.400, too.  Another
consequence of this general philosophy is that, unless it is encoded
somehow and mail viewed simply as a transport medium, the ODA-over-
extended-SMTP project becomes a non-starter.  And ODA-over-X.400 is
already being worked on by ISO, so maybe the "over the boundary"
principle should apply anyway.

Note that, for implementations that really intend to support 10646,
this approach is not significantly more difficult, or even
significantly different, in terms of implementation difficulties.  It
just makes things a lot easier for simpler-minded implementations.  And
I think it has an inherent robustness and clarity that 10646 lacks.  In
particular, "everyone supports 8859-1" is explicit in the 10646
proposals, it just arrives in a different form.

  ------------------------------

Part V.  What needs to be in the envelope? revisited.

  Assuming that extended SMTP uses the same port as the unextended
version (an option I favor for interoperability reasons, since I
suspect that most mail, at least in this country, will stay in ASCII
for a while, even if it is running over extended SMTP servers), the
envelope must have a verb to designate "8 bit data path" (or, if we
must, some variation on that which triggers unpacking, decoding, etc.).
If abandon the 8 bit data path, we get rid of that verb.  Ok.

The specific character set to be used (and I would argue that it should
be specifically designated even if the "10646" plan is adopted--sooner
or later, someone will want to send some character set that is
scheduled for the "next version" of 10646) does not need to be
designated in the envelope unless
  (a) Something besides the extended SMTP lingua franca is to appear in
the header (if the "everyone supports 8859-1" suggestion above is
adopted, then 8859-1 is as acceptable in extended SMTP headers as ASCII
is in the traditional ones.  We would, of course, want to review this
one a field by field basis).
 (b) The header does not have an internal mechanism for designating
alternate character sets.  The switching of 10646 or ISO2022 would
accomodate this requirement, subject to my hesitation about switching.
Ordering rules could be imposed that would specify character set prior
to whatever used (or might use) it.  Risto and I were playing with
syntax conventions similar to:
  Subject (8859-6):  abcdefghik...
for the extended arrangements.
   But I'd argue for the envelope anyway.  I think that MTAs do get
into the business of figuring out what is good for their UAs--our
layering is not that clean.  And I'd much rather put a little extra
into the envelope than have MTAs inspecting headers, which has, of
course, occurred in many situations on this and other networks.  If
people disagree, we get rid of that verb.

Finally folks want to embed things, with some sort of "Content type"
indicator, spelled in some special way.  Unfortunately or fortunately,
RFC822 was written very permissively relative to adding new headers,
experimental or otherwise.  If you want to add a new header, and give
it very precise semantics, you have to use one that violates some
existing rule in RFC822.  Other than the use of illegal characters in
field names, no such exclusion is possible.  So the obvious situation
is to invent another SMTP verb whose purpose is to say "message content
is not in RFC822 format, but RFC822++ format, where RFC822++ is an
extended *and restricted* form of RFC822.  Moreover, the arguments
above about wanting to reject, in an orderly way and at transfer time,
character sets you can't understand, apply as strongly or more so to
encodings, compressions, etc., that you can't understand.  So there are
arguments for the envelope containing warnings.  As an interesting side
effect, this solves the problem of the interaction of "old" mailers
that use dashed lines interoperating with things that assume, and give
strong significance to, an updated version of RFC934.... you announce
"extended SMTP" in the envelope and assert that the mail format
standard supported by "extended SMTP" is RFC822-extended +
RFC934-updated + fancy-character-sets.

   --john