Lois & Clark Forums Forums General Discussion Off Topic Spam Busters (Bayesian Filters)

As spam has increasingly been driving me crazy this week, I found this article on the BBC news website fascinating.

However, it and the related sites which I've tagged below quickly had my brain crying surrender.

So, for the more tech-minded out there - is there a way to translate this into something the technologically and mathematically challenged can actually use to decrease their spam count? Translations appreciated! (Hopefully in words of one syllable).

BBC News Website:

Unsolicited e-mails now infuriatingly clutter many inboxes, just as paper junk mail buried many a front door map. But is smart technology set to save us from spam?

To us humans, spam is very easy to spot.

Unfortunately to your computer one e-mail message looks very like another.

Without help it will see nothing special about the formatting in junk mail to distinguish it from the stuff you want to read.

Many anti-spam programs work by scanning e-mail messages for the keywords that spammers use, but your genuine friends tend to avoid.

But the spammers know this and use lots of tricks - some clever, some obvious - to fool the keyword spotters.

This explains the strangled spelling, strange spacing and replacement of some letters with numbers in words that the anti-spam programs are looking for.

"If you look at spam people hardly ever write the word Viagra anymore," says Paul Graham, a US software guru who has spent a lot of time studying junk e-mail.

Viagra often spelled V-l-a-g-r-a online
The tricks spammers use mean that keyword filters will only ever be able to stop a small proportion of spam.

They will always catch the obvious ones but, if the list of keywords is too large, they start stopping real mail too.

Mr Graham thinks that for many users an anti-spam system that stopped legitimate mail was far worse than one that let all the proper mail through plus a bit of junk.

"You definitely want to err on the side of conservatism," he says.

To do a better job of spotting spam, Mr Graham came up with a different technique that means he hardly ever sees junk mail anymore. "For me and all my friends spam is a solved problem."

The technique goes by the formidable name of Bayesian Filtering and uses probability to work out if a mail is junk or real.

Current versions are 99.7% accurate at spotting. Other Bayesian filters, such as CRM114, do an even better job.

This means that Mr Graham sees a couple of spams per week, instead of up to 150 every day without the filter.

The system is based around a huge corpus of junk and spam mails that Mr Graham gathered over a few months.

These thousands of messages have been statistically analysed to extract the top 15 features that define them as spam.

Any incoming mail is scanned to see how many of these defining characteristics it possesses.

The list of defining features includes some words, such as "teens", but others were less obvious and include formatting codes and routing information found in e-mail headers.

Mr Graham believes widespread use of Bayesian filters could destroy the spammers' business model.

The sheer number of spam mail sent means that even tiny response rates, reportedly 0.0001%, means junk mailers turn a profit.

"I think filtering 90% will probably be enough to do it," he said, "that would increase their costs by a factor of 10," says Mr Graham.

"Spammers are not really committed to being in the direct mail business."

Others are not so sure that the spammers will ever stop.

"It is like an arms race where the spammers come up with new tricks and people come up with a new way to detect them," says James Key, technology head at anti-spam firm Blackspider Technologies.

Mr Kay believes a combination of technology and legislation to make spamming illegal will be needed to beat back the tide of junk.

Certainly spammers must feel under siege at the moment.

US states are passing laws that outlaw spam, net service firms are filing lawsuits and installing basic filters. Some are even adopting Bayesian filters to spot the most obvious spam.

Who knows, one day soon spam might only ever be associated with processed meat.

Related sites:

A Plan For Spam

CRM114 - Best Bayesian Filter?

LabRat

I followed the links, Labby, with the intention of seeing if they have a downloadable version.

They do, but they've labeled it an "alpha" release. This makes me nervous (it seems akin to a warning, as do the number of bug fixes already added!); I'll wait until they have at least a beta release. But I approve of the idea, and hope it is successful.

Ooh, cool idea! I'm not too much bothered by spam, but the less I see, the better!

As for explaining it... far as I can tell, this is a computer program that can recognize the factors that make spam obvious on sight, based on the sorts you've already received. Sounds like it might easily adjust to any new tricks, too... I'd also prefer at least a beta version, though...

PJ

LabRat:
Bayesian Statistics is just a type of statistical analysis. So the guy collected all his spam and looked for common terms then wrote a program that will deny receipt of any message with those common terms. But since his spam would be representative of the populace of typical spam messages...er sorry, got off on my professorial horse. Anyway, it should work for anybody. But yeah, a beta version would be better. Could do bad things to your machine.
I don't know what they do, but Earthlink (my provider) has very effective spam control. We get next to none. http://www.earthlink.com/
I'm on a 56K modem too.

Artemis

Thanks, guys. I'll wait for a beta version then. Hope it arrives soon! Another round of strain on my delete finger this morning when I downloaded my first mail of the day.

Thanks for the link, Artemis - I'll check that out.

LabRat (still not boggled by the news from the experts that by mid-July they predict more spam will be sent across the internet than regular mail for the first time...)

And besides only being an Alpha version, at present they don't have anything that will work with Microsoft OS's. Mostly still Linux... so I think most of us are gonna have to wait a little while longer.

PJ

Hey, I guess we're to blame. This is from the LA Times.

May 27, 2003
E-mail story

American Spam Is Flooding Europe
[*] The EU's laws on junk e-mail are not as strict as those in the U.S., and enforcement is uneven.
By Matt Moore, Associated Press

STOCKHOLM ? The junk e-mail plaguing Europe has something decidedly in common with the American variety.

Nearly all the spam messages are in English, originate in the U.S. and don't even bother to price their wares in euros.

"It's always some unbelievable business opportunity, which is what we get from America," said Olle Thylander of the Swedish University Computer Network, a Stockholm-based group that oversees Internet traffic for Swedish universities.

Although there are no complete figures about the volume of spam Europeans receive, many contend it's on the rise. Yet Europeans find little common ground on how to combat the American scourge.

In essence, Europeans are importing an American problem but not any tough solutions. Unlike the strictest laws in Virginia and other U.S. states, European Union rules don't call for jail time.

At most, spammers are fined, but enforcement is left to individual countries. And although the EU rules cover spam sent from the United States and elsewhere, member countries lack the resources and the authority to pursue violators abroad.

"The main culprits are international companies, but we have no way of stopping them," said Cristina Garcia del Valle, a spokeswoman at the Spanish Ministry of Science and Technology.

Jens Storm Lernoe, who oversees Microsoft Corp.'s MSN and Hotmail operations in Denmark, estimates that more than 95% of spam comes from abroad, mostly the United States.

He notes that e-mail addresses are widely sold in the United States. Similar lists are bought and sold in Europe, but not to the extent that they are in the United States.

Spammers may be reluctant to base themselves in Europe because countries are smaller and it's more difficult to anonymously ply a trade that upsets so many people.

"There have been corporations here marketing via e-mail, but the negative feedback was so great they stopped," said Linnar Viik, a technology professor in Tallinn, Estonia. "You avoid the tools that will annoy your potential customers, especially when they may live nearby."

Europeans also tend to hold personal privacy more dearly than Americans.

"The Net is getting more mature, but more and more users here are feeling uncomfortable about it," said Gert Birnbacher, chairman of the Danish EBusiness Assn.

Many blame the Americans.

"It's not apparently from the United States. It is from the United States," said Mario Mariania of Tiscali, one of Europe's largest Internet service providers.

Unlike laws prevalent in the U.S., statutes in Europe generally require companies that send unsolicited e-mail to first seek permission unless they already have a commercial relationship with the recipient.

But enforcement varies.

"The directives say what the goal is, but the actual details are left to each member state," said George Mills, chairman of the European Coalition Against Unsolicited Commercial E-Mail.

Thus, there is no uniform way of punishing spammers, Mills said.

Fines in Italy average $280 for repeat offenders, for instance, whereas in Spain spammers can be fined more than $34,000, though that seldom happens.

In the first such ruling ever in Denmark, a court this month fined a Danish software company $2,200 for sending 156 unsolicited advertising messages. Some 50 consumers had complained.

Stefano Rodota, head of the Italian office that protects citizens' privacy, said his European colleagues "have said we need many more measures to prevent what is happening in the United States, so we are reacting."

The agency has filed criminal charges against compulsive spammers, but none have been convicted because, in part, of Italy's notoriously slow legal system.

Some countries simply have not made spam a priority yet.

In the Netherlands, Justice Ministry spokesman Victor Holtus said no laws or guidelines are being developed because the issue hasn't drawn many complaints.

In Germany, a planned June meeting of the country's Internet service providers will feature a session on technical improvements that companies can develop to block spam.

Harald Summa, general manager of Germany's Electronic Commerce Forum, said German ISPs have been trying to respond to consumer complaints and puts the blame on the United States.

His group has found that 35% of e-mail spam in Germany comes from the U.S., with an additional 7% from China.

"It's a problem of American marketing," Summa said. "It's so aggressive."
END STORY

Maybe that's why we have been looking out for how to winnow out received spam.

Artemis