Viewing all posts by Ian Lurie.

TF-IDF Killed The Copywriting Spam

TF-IDF score goes down as documents containing phrase go upNote: If you want to play with TF-IDF, download this spreadsheet. The first tab is a simple TF-IDF calculator. Enter the occurrences of a word, the words in each document, total documents and the number of documents containing the phrase. It does the rest. The second tab demonstrates falling TF-IDF as documents containing a phrase goes up.

Term Frequency-Inverse Document Frequency (TF-IDF) proves that, in the modern SEO game, quality trumps quantity.

TF-IDF is a text-mining algorithm …

Wait! Don’t run! I’m not here to teach you the math behind TF-IDF. Truth is, I barely understand it myself. But Term Frequency, Inverse Document Frequency (TF-IDF – a great phrase for the next SEO cocktail party you attend) contains some crucial lessons for us copywriters.

Here’s a very brief description of TF-IDF and how it works (a little fancy math involved):

TF = Term Frequency

We all know this one:

If our key phrase is “flibbergibbet,” and it occurs 4 times in a document that’s 400 words in length, then the TF for “flibbergibbet” is:

4 / 400 = 1%

Some folks call this keyword density. But we’re past that now.

Inverse Document Frequency

Inverse Document Frequency (IDF) is the inverse of the number of documents in which a phrase occurs. That’s a terrible description – I know that because the mathematicians I know all punched me in the arm after I said it. But it’ll work for our purposes.

In case you want to know:

IDF = log(total documents/number of documents with phrase)

So, if “flibbergibbet” appears in 250 out of 1000 documents, the IDF is:

log(1000/250) = .6


TF-IDF is the Term Frequency times the Inverse Document Frequency, or TF*IDF.

Here’s the thing about IDF that you must understand: As the number of documents containing a phrase goes up, the TF-IDF score goes down. Have a look at this graph — document frequency goes up as you move to the right:

tf-idf goes down as doc occurences go up!!

Yikes. So, the more times you mention a phrase, the less important that phrase appears on a specific page.

What It All Means

We don’t know for certain if the search engines use TF-IDF to determine the importance of a word on a page. But it’s likely they use it or something very like it.

Say you want your website to rank well for our favorite word. You include the word at least 3 times on every single page of your site. That actually reduces the TF-IDF score of each page for “flibbergibbet.”

Of course, there are many, many other ranking factors. Thousands. If your site is 150 pages of fantastic content, and it:

  1. Has a unique, fully descriptive title tag for every page
  2. Has a unique structure for every page
  3. Doesn’t spin or duplicate content
  4. Uses fully-descriptive ALT attributes, etc. etc.

… then TF-IDF probably doesn’t hurt you at all. A visiting search engine can use other signals to determine page relevance.

But content farmers, beware. If you crank out 999 pages of total crap, using your key phrase 5-10 times per page, all you’ve done is made it harder for a search engine to figure out which page is most important for that phrase.

If I were a search engine (and I’m not), I’d take that as a signal of a poorly-organized site.

Wouldn’t you?

The Lesson

In the past, site owners created page after page expounding on a specific key phrase, repeating it time after time in articles that were barely different, poorly written and poorly structured. That’s still a standard “SEO copywriting” tactic. I use quotes because it’s not SEO copywriting at all.

TF-IDF explains why that tactic has lost its power. It also shows why the cliche “If you want to rank, write good stuff,” really is the right strategy. TF-IDF means more isn’t necessarily better. So, write good stuff!

About Ian Lurie

Ian Lurie is Chief Marketing Curmudgeon and President at Portent Interactive, an Internet marketing company he started in 1995. Ian started practicing SEO in 1997, and has been addicted ever since. For more of Ian Lurie’s smarts, raves and rants, check out his Conversation Marketing blog.  He’s also published several reader-friendly, no-nonsense ebooks on SEO copywriting, including The Unscary, Real World Guide to SEO Copywriting and Fat Free Guide to Google Analytics. Follow him on Twitter: @portentint.

Just write! Ian Lurie on why you have to, even if you think you don’t

Guest Author, Ian Lurie

Ian Lurie is Chief Marketing Curmudgeon and President at Portent Interactive, an internet marketing company he started in 1995. Ian started practicing SEO in 1997, and has been addicted ever since.  As a steadfast fan of Ian’s candid and wicked-smart expertise, I’m delighted to feature his words of wisdom today.  – Heather

It amazes me how many business owners and employees think good writing is optional. Some of my favorite justifications:

“My customers don’t read”

Really? Do you sell to reptiles? Illiterate rodents? Tribbles, perhaps? ‘Cause otherwise, your customers read. Every day. Even if they don’t want to. They read signs. They read directions. They read e-mails.


If you sell anything, they read your e-mails regarding orders, sales, specials, etc.

If you provide a service, they read the reports and invoices you send ‘em.

And I guarantee they go to the Googles now and then to find an answer to a question.

Your customers do read. You’re confusing reading with reading books, or reading for fun.

The fact is, if your customers “don’t read”, then great writing is even more important. Your customers don’t want to spend a ton of time deciphering crappy writing. You need to get to the point with clear, directive prose. The best business writing goes unnoticed—all the reader remembers is the idea you were trying to communicate.

So, if your customers don’t read, you have to write that much better.

“My business doesn’t require any writing—it’s not that kind of company”

I. Uh.


I’m never sure how to answer that. When someone says that, my frontal lobe makes a kind of popping sound, and I black out for 30-45 seconds. Whatever happens after that must be bad, because when I come back to reality, the speaker looks like they got slapped with a rotten salmon.


If you make rubber grommets, you still need to explain why your rubber grommets are worth buying, right?

“Oh, but rubber grommets are a commodity. No one shops for them.”

Then you need to convince them to start shopping. That process starts with the first person who sees your site, if you’ve explained why you’re important to them. If you don’t bother, then yeah, they stop shopping, and go back to buying from the same place they always have.

There must be a reason you’re in the grommet business. If it’s just to sell to existing customers until they leave, I suggest you find a new line of work.

It’s up to you compel the reader. Saying “You’re not that kind of company” is just an excuse.

“My potential customers don’t use search engines, so I don’t need writing on my site.”

Surrreeee. It’s true. 10 or 12 North Americans out there never, ever search for stuff online. The rest of them, though, use Google, or Bing, or their computer’s search tools. Or, when they get to your site, they use your onsite search engine. Or, at a minimum, they’re going to search an individual page, with their eyes and brains, in an effort to figure out if they’re in the right place.

In all those cases, well-written copy that’s fully descriptive will help them find what they need. Which, in the end, is what turns visitors into customers.

At some point, customers search. Be ready when they do, with really good writing.

“No one’s going to read 500 words on shavers”

OK. So don’t write 500 words.

If you can present a compelling argument for your product in 25 words, do it. I don’t want 500 words where 5 will do.

“Writing” doesn’t mean “word diarrhea”. It means “communication”.

Yes, at some point, search engines get involved, and then it’s possible you’ll have to write more. That’s when you think about:

  • The questions you most often hear;
  • Special uses of your product;
  • Help/advice for best product use;
  • Your pet peeve (like people who think they don’t have to write);
  • Big changes in your industry that might have your customers wondering.

You can always drill deeper into a subject. There’s always someone who wants to read a 1,500 word treatise on the origin of the electric razor.

Just write!

No matter what business you’re in, you need to write:

  • Explanations of services, products, invoices, hours and policies.
  • Direction telling the reader what to do next, how to get help, etc.
  • Your case: Why customers should like you more than your competitor.
  • Communications with individual customers, via e-mail, Twitter, Facebook or whatever comes next.

It may be 10 words. It may be 10,000. It’s most likely somewhere in between.

Just write. The more you do it, the better you get at it. And, even if they don’t notice, your customers will appreciate it.

For more of Ian Lurie’s smarts, raves, and rants, check out his Conversation Marketing blog.  He’s also published several reader-friendly, no-nonsense ebooks on SEO copywriting, including The Unscary, Real World Guide to SEO Copywriting, and  The Fat-Free Guide to Google Analytics.