8 bit transport (RFC-ZZZZ) clarifications

[KLENSIN@xxxxxxxxxxxxxxx (John C Klensin)]

  Sorry about my silence of the last few weeks.  I was out of the
country for a bit under two weeks, then came back to 3/4 Mb of mail
about mail.  A bit overwhelming.  I'll try to hit the high points of
things that don't seem to be completely resolved.
  If I have missed issues that anyone raised and things are still
"active" (which is likely), please send me mail and I'll try to post
responses as soon as I can.

(1) EVFY.
  Just to close the loop on this, yes, it takes a mailbox as an
argument.  Its operation is intended to be identical to that of VRFY,
except that an affirmative response implies both that the mailbox
exists and that the host is willing to accept 8 bit mail for it.
Craig's message of 5 Jun 1991 10:31:47 -0400 (EDT) is correct in every
respect on this, especially the "If this isn't clear in the RFC-ZZZZ
document, it should be" part.
  However, the other implication of Philippe-Andre Prindeville's
question (again to reinforce Craig's response) requires non-ASCII
characters in mailbox names.  Neither RFC-ZZZZ, nor anything else I've
seen, makes provision for that.  Indeed, even if an RFC were written to
permit it, I'd suggest that the extremely narrow range of characters
permitted by X.400 in actual addresses (e.g., mailbox names) would
argue strongly against taking advantage of such a feature, lest one get
into gateway problems that would not be amusing at all.
  I do, personally, see a problem here (I like mailbox names that
reflect people's names, and it is unreasonable to suggest that people
spell their names incorrectly), but I think the place to solve it first
is in the X.400/MOTIS arena, not in the extended RFC821/822 one.

(2) Why bother with conversion delegation?
  In his note of the 29th, Erik did a good job of summarizing the
issues, following up Stef's earlier explanation which was actually more
clear than anything I've been able to put together.  Let me try another
slant, with the understanding that I'm probably not saying anything
new, and with the further understanding that they may not agree with
some of this.
  Like Greg (or at least my understanding of what Greg is saying), my
first preference under normal circumstances is that, if I try to send
something out in 8-bit form, and it hits a 7-bit "wall", that it just
bounce off and come back to me.  I think both the "enclave" discussions
and the suggestions of Bob Smart and others are complementary to this
notion: if one can avoid every hitting such a wall, or can avoid
hitting it except from the initial attempt of the originating MTA to
get mail off the sender's system, then we don't need *any* fancy
conversion models, specifications, or directions.
  I like that model a lot, but I've gotten into a lot of trouble on
these lists and elsewhere arguing that, if there were a lot more
bouncing and a lot less "fixing", the world would turn out to be a
better place.

  The origins of the notion of explicit delegation of conversion is
that we have observed four important phenomena on the Internet, and
especially with mail handling, in the last decade (things were quieter
before that):
  -- There is a lot of user pressure, to which sincere and clever
people have responded, to take things that are "invalid" and handle
them anyway.  This is a sort of "the mail must go through" principle.
  -- There has been a tendency for the authors of some mail software to
pass judgment on certain headers and formats, declare themselves to be
gateways (with all of the corresponding authority) and deal with what
they don't like by deletion or modification beyond recognition.
  -- There are gateways to other types of systems (some explicit and
some hidden behind MXs) that would like to Do the Right Thing, if only
they could figure out what it was.
  -- There are no protocol police--and protocol judges and protocol
jails--who have collections of conformance tests to help identify
violations of the rules and collections of sanctions to discourage that
behavior.
  These tendencies, in combination, predict that, no matter what rules
we write, there will be intermediate hosts who will accept mail in 8
bit and convert it to 7.  And they will justify that behavior on the
highest of moral principles.
  Because the folks in the first groups sometimes get it wrong (by my
definitions, if not by theirs), and because the last group does not
exist for me to appeal to, I want to assert control over my own mail,
to specify exactly what conversions get done and/or whom I trust to do
it.  Those two concepts can be inexorably intertwined, because I may
want to avoid hosts who don't perform standard-specified conversions in
the way that I read the standard.
  I want to stress that this model neither assumes, nor is a
preventative for, hosts who deliberately violate the protocols or
otherwise deliberately behave badly.  Other that the protocol police, I
see no solution for those cases.  What I do want to accomplish is to
give the hosts who will convert anyway, who want to do the Right Thing,
simple mechanisms for doing that which I trust.
  The important thing to remember is that, in this world of multiple
networks and protocols hidden behind MXs, multiple MXs for particular
sites, etc., I sort of launch messages onto the aether and hope for the
best.  In the newly-complete world we are creating, that makes me very
nervous. 
  Now, in that context, I don't see any particular problem with Craig's
suggestion about specifying an algorithm (or a specific conversion
rule, which may be the same thing).  I don't want to quibble about
syntax (let's see if we can agree on the principle, then start the
syntax process in a serious way), but I think it would be perfectly
plausible to say "ok, I authorize anyone to do this conversion who
claims to understand the registered (not 'standardized', Stef)
conversion model; if you need to convert and don't understand it, here
is the address of an agent that can perform the correct conversion".
In my capacity as a confirmed paranoid, I might never use this form,
but I see no problems with having it:  the corollary to wanting this
delegation to come from the originating UA is that I am perfectly
willing to let originating UAs make decisions that I consider stupid or
careless; I just don't want anyone asserting those decisions on the
UA's behalf because they think they are smarter and are trying to get
the mail through, and don't have enough information.
  And the "defaults" should remain Greg's defaults and my original
preferred defaults: it comes back to the sender, or it comes back to an
agent designated by the sender (probably on the sender's machine) for
processing.  On the other hand, Stef and I had an offline conversation
about "delegated enclave converters": a single agent that would take
all conversion-required bounces for an enclave and do something
enclave-standard to it.
  A few other issues were raised that deserve responses.  For example,
it was pointed out that all of this rerouting was likely to really slow
down the mail.  If it happened all the time, then certainly.  But the
idea is that 8 bit transport should be used to target 8 bit hosts.  If
that is what happens, then this mechanism "costs" one header line worth
of bandwidth, which is not a noticable slowdown.  A sender who is
interested in most-rapid transport will insure that the message only
goes to 8-bit-capable hosts, or will just send it in 7-bit form, or, at
worst, will specify conversion authorizations in ways that don't take
up much time.  A sender who is more interested in guaranteed accuracy
than speed will make different choices.  I think we should be in favor
of choices, rather than trying to second-guess applications not yet
invented.
  Ned raised the question of mail bouncing in and out of, say, BITNET
and possible conversion cycles.  I don't think there is an issue here,
unless it is an illustration of the fact that trying to bend the rules
to hide harsh realities from users gets one into trouble.  First of
all, there is the old protection of the wretched solution: once
something has been forced to 7 bits, no one is going to do any more
conversions of the 8 <-> 7 persuasion.  But, more important, Internet
messages go into BITNET in only two ways:
 o With FQDNs that point, possibly via MXs to a BITNET gateway and
host.  In this case, the gateways are presumably selected in a
reasonable way.
 o With user%bitnethost@gateway syntax, in which case the gateway is
explicitly chosen.
  and the come out, similarly, either specifying the gateway explicitly
or with FQDNs that identify (via MXs) gateways that the BITNET host
trusts.
  In any of these cases, the gateways are deriving gateway and
UA-conversion authority from the final destination, which, in my model,
is the other place it can come from.  They are "trusted" in that sense,
and there really is no issue about something coming out through a
trusted/ capable gateway and going back through an untrusted one.
  The only exception would arise if one had user@host.BITNET addresses
floating around the Internet, so that it was really the responsibility
of an Internet host to look at the form of the putative-domain and
figure out how to route (or rewrite) it.  But .BITNET is not a domain,
and those addresses are bogus if they appear on the Internet and I
don't think it is worth spending a lot of time worrying about them.
  It is also worth noting that *any* 8-bit message can be converted to
7-bit with absolute reliability (no information loss) by the simple
expedient of forcing it, probably (copies of) headers and all, into an
encapsuation in Base64.  I don't especially favor that as a sole
solution simply because I'm sure that the "do what the user really
wants" community will feel obligated, perhaps under intense user
pressure, to try to figure out when to use Base64 encapsulation, when
to use quoted-printable or quoted-readable, etc.  And it is that
precise decision-making pattern that causes me to conclude that I want
to put the decisions into the hands of the originating user.
  To put that differently, I don't want random, self-appointed
"conversion gateways" deciding what transformations lose information
and which ones don't.  Only the sender really knows the content of the
message.  I'd like to avoid conversions entirely.  If conversions are
to be performed, I'd prefer to see them universally distributed and
operating according to well-defined rules.  But I just read two
messages that said that such rules are adequate and better than this
"route to something the sender trusts category".  Interestingly enough,
one of those sets of comments was in a message that violated so many of
the (really very simple) RFC-822 rules that I'm mildly surprised that
it was delivered.  This is the issue: should I trust the implementation
on a host that can't get RFC-822 right to do conversions.  I don't
think I should be forced to do that just because it is unexpectedly
"on the way to" a host I'm trying to reach.

  Tbat is, in any event, why one would want this sort of thing.
Whether it is worth the trouble is another question.  As Stef has
pointed out, there are some potentially more interesting applications
for something like this than 8->7 conversions.   If someone could
promise that everything that was sent in 8 bits would either be
delivered in 8 bits or bounce--which seems to me the essence of Greg's
position-- then I'd probably conclude that this wasn't worth the
trouble.  But I just don't believe that.

Relative to lists and all of this, Kevin Donnelly writes, in part...
> Perhaps the best answer is to have upgraded list handling software
>which records against each member whether they can accept 8-bit mail or
>whether they require such messages translated to 7-bit.
  This certainly sounds like a sensible "list enclave" model to me,
even if, as Erik (?) pointed out, there are facilities in the RFC that
might avoid the need to do it this way.  Personally, especially with
large lists, I'm a firm believer in caches of information, however
those are organized.
      ---john