1. Introduction

As the Internet moves into increasingly commercial areas of use, an activity which entails considerable emotion and complexity is the use of direct marketing. The common term for this kind of activity on the Internet is "spam", encompassing commercial and non-commercial sending of bulk email.

The topic of unsolicited bulk email (UBE) has become highly emotional and political. Neither factor is conducive to careful discussion. The Internet Mail Consortium wishes to encourage careful deliberation in the public debate over this phenomenon and is developing a series of reports to facilitate that care.

Because of the great aggravation caused by UBE, thousands of people have devised methods to reduce it, both for themselves and others. These proposed solutions cover a wide range of ideas, and generally fall into two broad categories:

• Filtering
• Legal

Another way to look at this is to ask whether an originator is allowed to send UBE. If they are, the recipient or the recipient's host must filter; if not, then there must be some legal prohibition on sending UBE. The argument for requiring legal force is that it is necessary to counter the very strong financial incentive present for originators of UBE.

The distinction between legal solutions is, unfortunately, not clear-cut. Some of the specific solutions described in this report cross the categories; for instance, some of the proposed legal solutions rely on filtering as well. Still, these categories give a reasonable framework for comparing and analyzing the anti-UBE mechanisms.

2.1. Filtering

Most of the work in trying to stop UBE has been in the area of filtering. Filtering proposals can be divided into two broad categories:

• Heuristic filters (learning and guessing)
• Cooperative filters (definitive)

2.1.1. Heuristic filters

The majority of existing efforts to filter incoming UBE presume that it is possible to detect such messages without the cooperation of the originator. Some commercial packages for doing filtering have hundreds of rules. Unfortunately, originators of UBE are highly creative and are likely to develop sophisticated technical mechanisms to fool all but the most sophisticated filtering systems.

The heuristic approach fall into two categories:

• Origin filtering
• Message filtering

Origin filtering happens before a message has been fully received by the recipient's host computer, and is based in attributes such as the domain name and IP address of the originator. Message filtering happens after a message is received; attributes for message filtering can include the same as those for origin filtering, but can also include things like known keywords, "tip-off" headers, and so on. Examples of tip-off headers include headers that indicate that the message was sent in bulk or in a way that is hiding its true origin.

2.1.2. Cooperative filters

In addition to origin and message heuristic filtering for tip-offs that a message is UBE, it is possible to have collaboration between UBE originators and UBE recipients. Two entirely different schemes fall into this category:

• Content labeling
• Recipient registration

With content labeling, filtering is done by the recipient or the recipient's host; with recipient registration, the filtering is done by the originator.

For content labeling, messages can contain additional information supplied by the originator, such as the type of the content or an assurance of the originator's identity. When the originator cooperates by assigning useful labels, these messages can then be definitively filtered with full assurance that the filtering for those particular messages will be effective at preventing the UBE from getting to the recipient. However the recipient can also specify that they in fact desire labeled UBE.

Another proposal for reducing UBE, commonly called "opt-out", requires that the user inform either individual senders or keepers of universal lists that they don't want to receive UBE. This becomes a type of filtering, where the UBE originator filters who they were going to send to against a local or outside list of potential recipients. This type of originator filtering only works for honest and/or conscientious UBE originators, of course. Dishonest UBE originators would simply pretend to honor lists and then send the UBE anyway; other UBE originators would not even pretend to honor the lists. Because different forms of filtering can cause different negative side-effects, each type of heuristic filtering is described in this report separately. The sections later in the report describe what the filtering choices are made on, how effective that type of filtering is for blocking honest and dishonest UBE, the type of coordination that is needed for effective filtering across the Internet, the potential for information loss, and the impact of the filtering on recipients (which is, after all, the reason that anti-UBE tactics are being pursued.) Of course, the positive impact on recipients is not seeing unwanted UBE.

2.2. Legal

Many of today's UBE originators have shown little regard for honesty or normal business practices. Thus, many people think that the only way to prevent them from sending UBE, even if other anti- UBE mechanisms are employed, is through legal means.

For postal mail, direct marketing has relatively well-established practices, along with relatively long- standing regulation. However, UBE over the Internet is very new and is essentially unregulated. Of course, the Internet crosses many local, regional, and national boundaries, so any application of law to the Internet is sure to be problematic. This severely limits the ability to develop a coherent set of laws that can be applied to all (possible) source of UBE.

Legal controls are achieved through two different paths:

• Passing regulatory laws
• Signing contracts

To date, neither of these avenues is significantly exploited, so that specific alternatives are almost entirely theoretical.

2.2.1. Regulation

The nature of a law concerning UBE is to specify behaviors which are prohibited and provide for sanctions against violation of the proscription. Discussion of these kinds of controls focuses on two actors:

• Origination control
• Relay control

Control by the originator simply says that certain kinds of UBE shall not be posted to the mail system, or that certain kinds of UBE shall not be posted to certain kinds of recipients. Control by the relay says that the operator of an email relay service (including final delivery to the recipient) shall reject all or some UBE. A major difficulty with the development of either type of law is development of precise and useful definitions of the proscribed UBE. A particular danger is that a definition will cut too wide a swath. That is, an effort to control one type of content could turn out to include other, very different types of content.

2.2.2. Contracts

Origin-based mechanisms distinguish entire groups of UBE originators, usually everyone from a particular ISP or a particular domain. The rules used rely on originators to use the same or similar addresses each time they send UBE. Origin filtering prevents mail from UBE originators from being saved in the destination host's message store, and can reduce the load on destination hosts by reducing the amount of interaction they have with UBE originators.

The four methods described here are:

• Refusing IP connections from known UBE originators
• Refusing TCP connections from known UBE originators in the SMTP server
• Refusing SMTP messages from known UBE originators at the MAIL FROM command
• Refusing SMTP messages from originators whose domain name doesn't match their IP address

Note that it is possible to combine two or more of these techniques on a single server.

3.1. Refuse IP connections from known UBE originators

This technique, also know as "black holing", is based on telling routers at the local site to not route IP packets from a list of addresses that correspond to UBE originators.

3.2. Refuse TCP connections from known UBE originators in the SMTP server

Modern SMTP servers can be configured to look up the IP address or domain name of a originator as it connects to the SMTP server; the domain name is determined by a reverse lookup of the IP address. If the originator is on a list of prohibited sites, the SMTP server can refuse to accept any SMTP commands. Thus, the filtering is performed immediately after the TCP connection is opened, before any SMTP commands are exchanged.

3.3. Refuse SMTP messages from known UBE originators at the MAIL FROM

Modern SMTP servers can be configured to check the domain name given by a sending SMTP server during the MAIL FROM command in SMTP. If the originator is on a list of prohibited sites, the SMTP server can refuse to receive a message. This filtering is performed immediately after the MAIL FROM command, before any message is transmitted.

3.4. Refuse SMTP messages if the domain name and IP addresses do not match

Modern SMTP servers can be configured to perform a domain name search to find the IP address associated with the domain name, then check if that address match the IP address of the TCP connection. If the two IP addresses don't match, the SMTP server can refuse to receive a message. This filtering is performed immediately after the MAIL FROM command, before any message is transmitted.

4. Message-Based Heuristic Filtering

These mechanisms attempt to distinguish special content, tell-tale headers, style of addressing, and so on, for partitioning regular mail from UBE in the recipient's own environment. The rules for filtering can include domain names and IP addresses, similar to the coarse filtering described earlier.

Specific filtering occurs in two places:

• Message store (before the recipient has retrieved it)
• Recipient's mail client

The main difference between these two is that filtering in the message store usually removes the message from the message store or labels it in a way that the mail client can recognize.

Implementations of specific filtering must decide what to do with messages that get caught by the filter. If the filter simply throws out the message, it runs the risk of losing non-UBE that simply looked like UBE, or throwing out UBE that the recipient actually wanted. Even if the filter doesn't throw out the UBE, but merely sets it aside for later reading by the recipient, misfiltered mail can easily get lost among the UBE that it appears in.

4.1. Filter messages in the message store

A receiving host can scan each message and filter out suspected UBE. It can also periodically scan the message store, for example when it has updated its list of rules about what might be UBE, to see if any of the received but unretrieved messages should be marked as UBE.

4.2. Filter at the mail client

A mail client (such as one using POP or IMAP) can scan each message as it comes in from the message store and filter out suspected UBE. This happens as the recipient is receiving messages.

5. Enforcement by Regulation

To date, none of the heuristic filtering mechanisms have enjoyed enough use to afford recipients with much freedom from UBE. Because many of these filtering mechanisms have significant side-effects, it is unlikely that they will have long-term effectiveness, in spite of their wide appeal. Even if the general populace of recipients and receiving hosts decide to use heuristic filtering, UBE originators will simply work harder to get around the filters and are likely to succeed.

Legislators, most notably in the U.S., have taken notice of the UBE problem, and laws have already been passed in some locations. In order to be effective, the laws must define and proscribe specific behaviors. This requires that the legislators understand the technical nature of both the problem and the enforcement of the laws, and that the enforcement agencies understand and have the ability to prosecute transgressors. To date, many people are skeptical that, even if well-crafted laws could be passed, that they would be enforced with enough vigor to deter senders of UBE.

The biggest problem associated with legislation about UBE is the fact that the Internet by and large ignores political boundaries. For instance, some states in the U.S. are already considering (and, at least one case, passed) legislation regulating UBE. However, intrastate communications pretty much ignore local content-specific rules. The same is true for national laws: the content on the Internet is global and it would be impossible to limit originators based on their location.

5.1. Types of anti-UBE laws

Because UBE is fairly new and most legislatures are fairly slow, there is little experience with anti- UBE laws. To date, most of the legislation has been in the U.S., and at the time this is being written, no national laws have passed.

The proposals fall into two categories:

• Outright bans
• Mandatory labeling

The latter is particularly worrisome to many people because it means that specific heuristic filtering must be in place in order for the law to have any effect; users without such filtering would likely receive even more UBE than they do now. Many also worry that once we have mandated content labeling for some Internet traffic, labeling other types of traffic is sure to follow. Also, such mandatory labeling doesn't do anything to reduce the cost on relays or recipient hosts.

Even in the case of outright bans on UBE, it is expected that there will be protracted lawsuits about the definitions used in the laws. Defining "bulk" and "unsolicited" will certainly entail compromise, and it's not clear how much UBE will be reduced in the long term if today's fly-by-night UBE is prohibited but more "legitimate" originators step in.

5.2. Enforcement of anti-UBE laws

There are two approaches to the enforcement of laws. One has agents who prevent the occurrence of the offense and the other is to apply a penalty to those who transgress. For example, in automobile speed control, the individual driver is responsible for prevention, with the police detecting transgressions.

For Internet UBE, there appears to be little hope of having a broad base of conformance by individual originators. This has led to an effort at making relay service providers responsible for detecting and preventing the transmission of UBE.

The difficulty with relay control is both technical and social. The technical problem involves the difficulty of an email relay service provider in distinguishing one type of email from another. The social difficulty is with having an email service provider inspecting each user's messages. In many jurisdictions, this is considered to be an invasion of privacy and may be illegal.

When exploring new legal territory, as with development of new engineering, the process is often facilitated by re-using well-established practice. In looking for an established law which might translate well to the world of UBE, perhaps the most obvious example pertains to U.S. regulation of the sending of unsolicited advertising to telephone facsimile devices.

The premise behind the fax control is that unsolicited advertising consumes the recipient's resources, both scarce communication service and costly ink and paper. At first it might seem that email is subject to neither concern, yet there is strong indication to the contrary. Current email handling technology for users does not allow very high quality separation of UBE from other email, so that the recipient must spend significant time doing the separation by hand and, in the process, there is a tendency to lose non-UBE mail. Further, many users pay for their email access and UBE costs them money. An extreme example are researchers at sea who pay very high satellite charges for email access and for whom UBE represents an extremely damaging drain on the budget.

A contentious issue being faced by legislators who want to control UBE is the question of jurisdiction. The U.S. is typical of many countries in that it has different enforcement agencies for commercial laws (the Federal Trade Commission) and communications laws (the Federal Communications Commission). Should the anti-UBE laws be enforced as commercial acts or as communications?

Not all legal remedies have to come from legislation. For example, most ISPs in the U.S. require their customers to agree not to send UBE from their service. These agreements, while legally binding, are often difficult to enforce due to the requirement for reliably identifying originators of UBE and in proving the level of damage to the ISP. Even so, some companies are actively pursuing litigation against UBE originators that violate their license agreements.

Some UBE solutions have called for standardized labeling of UBE. The obvious requirement for labeling UBE at the source is cooperation by the originator. This might come from legal restrictions on UBE, or it might be voluntary, showing that a originator is being honest about their mail as a way to generate good will among recipients.

Labeling alone is not sufficient to reduce UBE; it must work in coordination with filtering. However, labeled mail is significantly easier to filter than unlabeled mail, since the filtering software can use definitive rules to check for labeling. More importantly, labeling permits accurate, definitive separation of UBE from non-UBE. There are two proposed methods for labeling UBE:

• Object
• Channel

The first attaches labels directly to the messages, independent of its means of carriage. The latter identifies different data conveyance mechanisms or paths for different types of data. For example, non-UBE might be sent to the usual SMTP TCP "port" and UBE might be sent to a different port.

The kind of additional information that can be added by the originator falls into two categories:

• Labeling for the type of content
• Assurance of the originator's identity

Explicit labeling that the content is UBE makes filtering for UBE a simple task. The originator either uses a standard method to state in the message that it is UBE, or sends the UBE using a different mail protocol that is reserved for UBE.

Another method that aids definitive filtering is giving the recipient assurance that the originator is who they claim to be. Just as anyone with access to a mailbox can send postal mail, anyone with an Internet connection can send Internet mail. Accountability is desired because, in both systems, full accountability of originators would reduce the amount of bothersome mail being sent because recipients could more easily find the originator to complain. Accountability is discussed in Section 9 of this report.

7.1. Object labeling

Labels can indicate the "class" of the message, such as "commercial", "unsolicited", "for adults only", and so on. Proposals for this kind of labeling have almost all been within the RFC 822 headers for the entire message. Some proposals have been for new headers, while others have proposed reuse of common headers like "Subject".

Adding content labeling to distinguish individual components of the body of Internet mail messages could be difficult for messages which have a structure that does not allow easy modification for labels. On the other hand, messages which have their content structured using the MIME standard could employ a envelope header such as "Content-Label" to distinguish the nature of individual components of the message. Although it permits very fine-grained labeling, this approach would be expensive to employ for receipt-time filtering, since it requires thorough inspection of the entire message before deciding upon its disposition.

With either new headers or re-use of current headers, all filtering software must be aware of the rules used in labeling. This means that recipients who do not have filtering performed for them will receive all UBE.

7.2. Channel labeling

There are many ways to send a message. Some proposals would force (or simply allow) UBE originators to use a different TCP port for UBE than is used for normal Internet mail. Because this port would be reserved for UBE, a receiving host could decide on a case-by-case basis whether or not to accept the message for one or more of the recipients who use that host.

This type of labeling by delivery channel opens up interesting possibilities. If UBE is legally banned for normal Internet mail channels, using an alternative channel allows recipients who want to receive UBE to get it. Recipient hosts could choose whether or not to offer UBE receipt as a service to their clients. In fact, recipient hosts could make arrangements to only accept UBE that they have been paid to accept; they could then keep the money or even share it with their customers.

7.3. Choosing labels

Most proposed labeling requires the originator to determine the class of the message. Unless labeling is required by law, there is no one to adjudicate whether a particular originator mislabeled or failed to label a message properly.

The term "commercial" could have different meanings in different places. In some places, offering something free to someone has a different meaning than offering to sell them something, even if the free item or service is a lead-in to a commercial appeal. A significant portion of the UBE at the time that this report is written consists of statements to the effect of "come to this Web site". Since that act doesn't cost the recipient anything, some might not consider that "commercial".

Before it was clear how pervasive UBE would become, some people proposed there be a simple way for people to ask not to receive any more UBE from a particular originator; they believe that this would greatly diminish the problem. In the world of postal and telephone direct marketing, such explicit registration is called "opt-out" and is reasonably effective. The established direct marketing world primarily relies on a few centralized services which register people who opt-out, and these services then furnish the list of such people to marketers.

Proposals to have potential recipients register for Internet mail opt-out have varied between:

• Source registration
• Clearinghouse registration

The former entail having each potential recipient contact each potential source of UBE, to request that they not send UBE to them. The latter has the potential recipient contacting a (presumably small) set of independent services which compile "opt-out" lists and furnish the lists to senders of UBE.

Both proposals have been shown not to work. Unscrupulous UBE originators have not only not honored individual requests from users, they have sold lists of the people who responded to other UBE originators as "people who at least read their mail". Other unscrupulous UBE originators who say that they participate in opt-out clearinghouses still send mail to recipients who have registered with the named clearinghouses to not receive UBE. Because these kinds of clearinghouses are voluntary, there is little incentive for a UBE sender to use them.

The requirement that a recipient should have to respond to anyone who sends them UBE does not scale well with the large number of potential originators. Each month, hundreds of different originators send UBE, and that number could easily become tens of thousands in a short period of time. Forcing Internet users to reply at least once to that many mailings is unworkable.

In the early days of the Internet, many people knew each other, or knew at least one person at most Internet sites. This caused people to treat each other with a fair amount of respect, since they could most likely be held responsible for their actions. Of course, the Internet is not like that today.

One way to bring a bit more accountability back to Internet mail is to force either users or mail originating hosts to certify who they are. This gives a recipient a definitive method for finding the originator of a message, which is particularly important for eliminating UBE.

In order for most originators to want to start using accountability methods, a significant number of receiving hosts or recipients must filter for messages that aren't accountable. This filter would flag messages that were accountable so that the recipient might trust them more, or possibly read them before messages that weren't accountable. Without this kind of pressure, there is little incentive for originators to add the overhead of accountability.

Accountability for Internet mail falls into three broad categories:

• End-to-end authentication
• First-hop accountability
• Relay accountability

9.1. End-to-end authentication

Secure email can be used to certify that the person sending a particular message is indeed who they say they are, assuming that both parties agree on particular arbitrator of authentication. Each message can be tagged with a certificate that is digitally signed by a trusted third party. These certificates contain unforgeable credentials, so that the recipient can trust the identity of the originator as much as they trust the certifier. These certificate authorities (CAs) might be a bank, a national postal service, or a private entity.

This type of authentication only works in a system as large as the Internet if there is a small number of CAs. If there are more than a few, then the authentication portion of the message can become quite long, and the processing time for verification of the authenticity also gets longer.

Further, there must be some persistence in the authenticated information. If an originator is constantly sending UBE under a different name, then the goal of locating them and resolving complaints will not be served.

9.2. First-hop accountability

A modification to the concept of end-to-end authentication is to only authenticate the SMTP server that injected the message into the SMTP stream (at the "first hop"). This kind of signature can be done as the message is being processed. In order for this mechanism to work, all first-hop originators who become certified must agree to not accept mail from anyone they do not know. This system entails two contracts: between first-hop server owners and their customers, and between servers.

CAs would only issue certificates to sending hosts which commit to honoring first-hop accountability. Because this belief might change over time, the certificates would probably have shorter lifetimes than certificates for end users. If a significant number of the certified hosts cheat, the system would quickly fall apart.

In all likelihood, this approach will use a hierarchy of trust which begins with ISPs who commit to enforcing a set of rules for their customers and provide service only upon obtaining contractual commitment by those customers. Hence a violation of the terms and conditions of that agreement will provide a basis for legal action against the offender.

First-hop accountability also requires that the originating host have a definitive way of knowing that the person who sent the mail is who they say they are. There are many methods of doing this, including checking the login information from the dial-in server, using domain names that are controlled by the owner of the sending host, or requiring login to the SMTP server. This last type approach is not typically available, although mechanisms for SMTP "login" are being deployed.

If a recipient has the ability to distinguish services that enforce accountability, recipients can filter out sources which do not provide the enforcement. Services which can identify their users during message posting can hold them fully accountable by requiring that users sign an agreement which specifies particular rules concerning UBE. If the user violates those rules, they have violated a contract, and the usual legal recourse is then available.

9.3. Relay accountability

Growth of UBE in the Internet is having an extremely negative effect on email use. Originators have no incentive to limit their transmissions and recipients have inadequate tools for distinguishing legitimate email from UBE. They also have no meaningful legal recourse.

In this report we have described the various technique currently being discussed for the control of UBE. The taxonomy of choices described in the report includes:

Filtering
      Type
            Heuristic
                  Origin
                        IP host or network address
                        Domain name or domain suffix
                        Mail-From
                  Data strings (headers and content)
            Cooperative
                  Recipient registration
                  Originator labeling
      Location
            Source
                  Opt-out and opt-in lists
            Relay/Recipient SMTP server
                  Origin information and/or data strings
            Message store
                  Origin information and/or data strings
            Recipient user agent
                  Origin information and/or data strings
Law
      Regulation
            Ban on sending
            Ban on relaying
            Mandated content labeling
      Contracts
            Originator prohibition
            Relay associations
            Voluntary content labeling
Accountability of originator
      End-to-end authentication
      First-hop accountability

None of the proposed solutions to the problems associated with UBE will work in all environments. Many of the technical solutions will alleviate some of the problems today, but do not adapt well to smart, motivated, uncooperative UBE creators. Heuristic approaches to control of UBE have the serious side-effect of silently losing some legitimate Internet mail. This effect, even in the name of preventing UBE from getting through, is unacceptable to most recipients. Of course, it would be best if the Internet mail community could devise technical means for dealing with the issue instead of resorting to the often-problematic legal system. To date, neither type of solution is the clear-cut answer to the problems being faced by Internet users.