Re: Comcast blocking mail to its customers

[David Farber <dave () farber net>]

Begin forwarded message:

From: Gordon Peterson <gep2 () terabites com>
Date: October 16, 2008 1:02:57 AM EDT
To: dave () farber net
Subject: Re: [IP] Re:       Comcast blocking mail to its customers

David Farber wrote:
> Begin forwarded message:
> From: Tilghman Lesher <tilghman () mail jeffandtilghman com>
> Date: October 15, 2008 7:32:17 PM EDT
> To: johnl () iecc com
> Cc: dave () farber net
> Subject: Re: [IP] Re:      Comcast blocking mail to its customers
> On Wednesday 15 October 2008 15:32:50 David Farber wrote:
> > Begin forwarded message:
> >
> > From: John Levine <johnl () iecc com>
> > Date: October 15, 2008 3:12:36 PM EDT
> > To: dave () farber net
> > Subject: Re: [IP] Re:      Comcast blocking mail to its customers
> >
> > > My view is that an appropriate AUP for email should be similar to
> > > that of a common carrier or the USPS.  It's a critical service these
> > > days.  Using robotic methods or wholesale IP shutoffs to dump
> > > presumptive spam into the trash is not acceptable for such a
> > > service.
> >
> > The mail stream that ISPs see is typically 95% spam these days.  That
> > means 20 spams for every real message, so if they were to accept and
> > store all the spam, that's more than an order of magnitude increase  
> > in
> > the size and cost of their mail system, which would be passed through
> > to the customers, most of whom don't want it.

While it's certainly true that we don't want spam in our inboxes, the  
argument collapses since the mail system tends to be such a small  
percentage of the total ISP bandwidth, compared to Web (and torrent,  
video, etc) consumption.  Likewise, disk storage has gotten cheap  
enough that E-mail just doesn't take much, as a percentage of what the  
user generally pays per month.

>> ...And even if they did,
> > how much confidence do you have that you could manually sort it
> > correctly?  I've seen plausible studies that say that mechanical
> > filters are if anything better than humans at sorting large mail
> > streams, since mechanical filters' eyes don't glaze over.

First of all, the DEFINITIVE definition of what a given recipient  
considers to be spam is the RECIPIENT, and not the recipient's ISP  
(nor some other ISP upstream).

> I think you missed the part which I consider to be most important,  
> that of
> dumping presumptive spam into the trash.  The most correct method of
> filtering is to do it at SMTP time and reject the email then, rather  
> than
> trying to either a) accept all email and bounce the stuff that is
> undeliverable (this is arguably what is most wrong with some MTAs,  
> such
> as qmail, as it causes the secondary problem of backscatter) or b)  
> accepting
> all email and tossing the stuff that a mechanical filter thinks is  
> spam
> (which means that a sender may never be notified that their message  
> was
> falsely flagged as spam).

Filtering at SMTP time as you are discussing is only truly viable (as  
for the benefits you're claiming) in cases where no forwarding is  
taking place.  Otherwise, an SMTP reject only is immediately available  
to the one-hop-upstream relay, and if forwarding is taking place, the  
original sender is perhaps already long gone.

I will also point out that sending back a "bounce" message for spam is  
a TERRIBLE approach, for a whole variety of reasons:

  1)  it results in multiplying the bandwidth wasted by the spam;

  2)  If mail headers are forged, as spammers often do, it is  
practically impossible to determine who the real original sender of  
the message was.

  3)  That fact means that spammers could (and do) send OBVIOUS spam  
to users at ISPs who bounce the spam back... and forge the return  
address and Received: headers such that the bounce goes back to the  
spammer-intended recipient... the spammer bounces the spam off the  
rejecting ISP, using it as an unwitting (and less likely to be  
blacklisted) relay.

  4)  Yahoo, in particular, is very bad about turning off mail  
delivery when it sees a "hard bounce" implicating that e-mail  
address... even when the "hard bounce" is bogus and comes from a  
stupid anti-spam filter dealing with a message with forged headers.

As inelegant as it admittedly is, the best way to deal with spam is  
probably just to "black hole" it.

> Rejecting at SMTP time guarantees that minimal backscatter bounces are
> generated

Bounces are NOT a good idea, for reasons including those mentioned  
above.

> ...and when an email is rejected as a false positive, the sender has
> immediate feedback of the problem and can work to address the issue.

Ultimately, the only person who can TRULY decide if mail is spam or  
not, by THEIR definition, is the RECIPIENT.  The best solution is to  
take questionable cases and to at least ALLOW the recipient to make  
the final call.... and to decide whether they want to get back to the  
sender and advise them of a delivery problem.  This can eliminate the  
"false positive" bounce issue.

> It is really no more computationally expensive than current  
> operations (which
> have to scan all email anyway, so they might as well do it at SMTP  
> time).

That's not really true, since doing it at SMTP time means that you  
have to identify and deal with spam in real time.  Not only does that  
increase SMTP latency, but (more critically) it prevents approaches  
which recognize spam floods only after a large portion of the flood  
has already arrived and "passed" preliminary processing.  It can  
provide better identification and handling if one can catch spam  
somewhat later in the process, after the characteristics of the spam  
flood are by then better understood.

> ...In
> the case of a flood of email causing problems with scanning (the prime
> argument against scanning at SMTP time, that it does not scale),  
> that is
> easily addressed within the mail protocol, simply by sending a 400- 
> level
> error, indicating a temporary issue, which good MTAs use as an  
> indication to
> try the delivery again later.

That doesn't really change the problem very much.  You are still  
complicating the mail transfer process, and adding to the percentage  
of the mail system loading that must be done in real time.

> Oddly, a 400-level error stops many spam bots
> in their tracks, which will never reattempt delivery of the same  
> message upon receiving the first error.

Such a strategy is not durable, because spammers can and do change  
their  algorithms based on the way recipients handle the spam.  There  
is little point in implementing convoluted approaches which the  
spammers can evade more easily and faster than it is to construct the  
defenses.

One simple and remarkably effective first-stage rule is to allow the  
recipients to simply declare that BY DEFAULT they wish to block mail  
from recipient-untrusted (or unfamiliar) senders if the message:

  1)  contains HTML (key to all manner of evasion of antispam  
filtering);

  2)  contains any attachments (likewise, and key to most viruses and  
worms arriving by E-mail... just today alone I received two spam  
messages with apparently hostile attachments);

  3)  is bigger than a specified size.

I believe that recipients should be able to establish a fine-grained  
"permissions list" of people who they accept mail from, and what TYPES  
of mail they expect to see from that sender (for example, I'm not  
going to accept executable attachments from most folks, although if my  
dear old Aunt Gertie wants to send me a picture of her poodle Fifi,  
that's probably safe).

Establishing such a default rule for previously unfamiliar/untrusted  
senders means that senders would need to negotiate in advance with  
recipients (and using small plain text messages) if they want to send  
HTML mail or attachments (or large messages, for that matter).  Of  
course, that's only just courteous behavior anyhow... although it  
would be nice to have that practically become A REQUIREMENT, by  
convention.

This default rule by itself would virtually eliminate E-mail as a  
transmission vector for viruses and worms... so it helps with a lot  
more than only just the spam problem!

--

Gordon Peterson II
http://personal.terabites.com
1977-2007:  Thirty year anniversary of local area networking




-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com