Content-based Spam filtering - losing battle?

Quote:

Original post by Dmytry It seems to me that spam filters(not to be confused with blacklists and whitelists) could not keep up with latest spamming methods and are going to be defeated in foreseable future, or the rate of false positives (non-delivered emails) is going to increase. ... So, are we doomed? discuss.

I doubt it. A while ago I registered a spam e-mail address @gmail.com and deliberately posted it around a few forums. I also signed up for a few mailings as a test case. Within about 72 hours I had my first unsolicited e-mail, but it went straight to the Spam folder. Google correctly identified every message that was spam.

It did have one false positive, an e-mailed weekly newsletter from Men's Health. I suspect it tripped up over the aberrant concentration of sex-related articles ("Boost Your Libido!", "How To Rev Her Up In Bed", "Herbal Supplements -- Quackery or Quality?") that were in that particular newsletter, since it didn't block the others from the same series. (I also took the step of clicking on the letter and marking it as "not spam", which probably strongly weighted the decision about whether the next letter was spam.)

Quote:

Automatic "training" only makes things worse. I do not see how anything could be done with spam that (for example) contains large randomly selected piece from classic literature (or wikipedia, or whatever else), with few links in the middle.

Context, of course. The probability that a link to Viagra will occur in the middle of an actual Shakespearean soliloquoy is virtually nil. "To be or not to V1AGR4, that is the question..."

I've been using POPFile in Outlook and it's given me great results. It's a bayesian filter so you have to train it but it doesn't take as much work as you would think. Since Oct 8th I've gotten 56,304 email messages, and POPFile filtered 94.15% accurately. Works well enough for me...

I've never recieved one single spam mail on my hotmail account (which I use for various stuff where I dont want to give out my normal mail address). I have, however, had problems with it dumping legit mails into the spam folder (and the spam folder is emptied every day). So now I've set the spam filter to "low". Still no spam though, the only difference is that hotmail keeps bugging me about "your spam filter is set to low blah blah".

I'd like some sort some sort of whitelist/blacklist based on digital signatures... So that signatures which you trust will never be counted as spam. The problem here is that not many people actually use digital signatures [sad]

I think the future of spam filtering is through networking. People are the most effective spam filters in existence at the moment. If you can detect when a group of messages share similar characteristics (e.g. all start with broken text and then continue with correct english, all have a particular link or image attached), and you can recieve reports from some users that the messages are spam, then you can junk the messages for the rest of the users.

I have a sneaking suspicion that this is what GMail does.

Quote:

Original post by Dmytry It is obvious that naive techniques such simple sum of "spammness" indexes assigned to different words (so-called "Bayesian filtering") can not keep up even with most basic techniques such as addition of many random words.

Actually, Bayesian estimation is not the problem, it's the way it's used. Bayesian estimation is the basis for the whole of formal estimation theory, control theory, large parts of robotics, and other fields of engineering. It is a very powerful tool because it lets you quite literally get information from "nothing".

You are correct, though, that it is difficult when just applied to get the "spammness" of each word, since each word by itself does not constitute a "spam" message. Unfortunately, there aren't many better ways to apply it to spam filtering.

I just use a simple ruleset run one at a time in order.

1. Move all mail (that just came in) to junk folder.
2. Move all mail adressed to me and others to MassMail Folder.
3. Move all mail adressed just to me to Personal Folder.
4. Move all mail for (adress list) to Personal

I end up with very few letters I want in junk, maybe one a month, and very little junk, maybe four or five a month.

I use Thunderbird's built-in baysian filter. I get quite a few mails. On average I end up with one spam message in my inbox. The last time it marked a legit message as spam was in the week I installed it (when the filter wan't trained yet).

What really surprizes me is that there's still a large ammount of spam sent out in the first place. Paying a hacker with a botnet to send out a spamrun of a million or so can't be cheap so the ROI must be pretty good on spam runs.

Quote:

Original post by kSquared Quote: Original post by Dmytry It seems to me that spam filters(not to be confused with blacklists and whitelists) could not keep up with latest spamming methods and are going to be defeated in foreseable future, or the rate of false positives (non-delivered emails) is going to increase. ... So, are we doomed? discuss. I doubt it. A while ago I registered a spam e-mail address @gmail.com and deliberately posted it around a few forums. I also signed up for a few mailings as a test case. Within about 72 hours I had my first unsolicited e-mail, but it went straight to the Spam folder. Google correctly identified every message that was spam. It did have one false positive, an e-mailed weekly newsletter from Men's Health. I suspect it tripped up over the aberrant concentration of sex-related articles ("Boost Your Libido!", "How To Rev Her Up In Bed", "Herbal Supplements -- Quackery or Quality?") that were in that particular newsletter, since it didn't block the others from the same series. (I also took the step of clicking on the letter and marking it as "not spam", which probably strongly weighted the decision about whether the next letter was spam.) Quote: Automatic "training" only makes things worse. I do not see how anything could be done with spam that (for example) contains large randomly selected piece from classic literature (or wikipedia, or whatever else), with few links in the middle. Context, of course. The probability that a link to Viagra will occur in the middle of an actual Shakespearean soliloquoy is virtually nil. "To be or not to V1AGR4, that is the question..."

But how you can automatically determine context? Have database of all literature, encyclopedias, webpages, etc? Isn't it going to have insane false positive rate?

I'm talking of spam like this:

To be or not to be, that is the question. Whether 'tis nobler in the mind to
suffer The slings and arrows of outrageous fortune Or to take arms against a....
Of course real spammer aren't going to use well known texts, he could use random pages from the web.

Also: there is still a possibility for large email providers to use non-filtering approach, such as blocking links that are sent to too many users (sort of blacklisting), or not letting same IP to mail too many users. You can't say for sure that google blocked emails using content filter... Google has a lot of other information avaliable besides content.
Another possibility is to crawl the page linked from email and use content from there, this indeed could work even if email looks okay.

[Edited by - Dmytry on December 6, 2005 3:11:26 AM]

Quote:

Original post by Sander I use Thunderbird's built-in baysian filter. I get quite a few mails. On average I end up with one spam message in my inbox. The last time it marked a legit message as spam was in the week I installed it (when the filter wan't trained yet). What really surprizes me is that there's still a large ammount of spam sent out in the first place. Paying a hacker with a botnet to send out a spamrun of a million or so can't be cheap so the ROI must be pretty good on spam runs.

so called "bayesian filter" is just having database of "spamness" of individual words, and computing sum for all words in text. It's the first idea that could come to programmer's mind when he sees spam that mentions viagra alot. Really, he doesn't even need to know about bayesian filtering in first place. Bayesian filtering is really going to die; most spam i see includes random text and therefore can't be filtered this way unless you're okay with many false positives. Bayesian filtering is linear, and to really judge things by content and filter emails containing random words(with probabilities same as in real text) you must use something non-linear.
Bayesian filtering can be used where it is apporiate: when different items are independent. But it definately is not true for texts. Of course it helps that most spammers are complete morons and can be keept out even by simplest filtering, but they do learn from better spammers.

edit: also, with personally "trained" bayesian filtering and random stuff in email you will get to point when only topic you usually discuss gets through, and any other topic will not. (your common words will get high positive weights and everything else will get negative)

edit:
By the way.

Note: i'm getting spam only on one of my "public" email addresses. Of course for more serious stuff i use private one that receives absolutely no spam..

[Edited by - Dmytry on December 6, 2005 3:13:17 AM]

Wanadoo's spam filtering is unreliable. It'll suddenly decide to class people as spam after I've received and replied to their emails. For instance modellers I've been working with have suddenly stopped sending me email, but actually they were suddenly spamming me.

And the latest GD newsletter was also flagged as spam. I can correct the system but why does it do this? Surely if I reply to an email it should take note of that?