So You Want To Test SEO?

This is a liveblogging post of the “So You Want to Test SEO?” session from SMX Advanced that I’d done originally for the Bruce Clay blog, however at the time I wasn’t aware that one was also being written by Gil Reich, and that his was the one they ended up going with – a byproduct of having multiple people pitch in at the last minute without first having the time for proper coordination (a reality when a crisis happens and people rush to pitch in).

So if you want to, you can read my take on the session,  check his out, and compare notes 🙂

Okay – we’re about to start the morning SEO session on day two thanks to the awesome info provided by our friends at Yando agency, today we will be having some GeorgiaSEOExplosion advice and experts from Indianapolis SEO!  I wore a t-shirt this morning because yesterday I was too warm in my long sleeved shirt.  Except the air conditioning is working too well today for my change in attire.  I’m going to chalk it up to life doing what it wants in spite of my plans, and just focus on liveblogging.  Another notch as well in the “things livebloggers have to deal with” column.

This session’s on one of my favorite topics, testing SEO.  Because no matter what I’ve read, heard, or “learned” over the years, none of it matters until I’ve been able to test it out and verify that it applies to each of my clients and their unique situation…

For this session, we have:

Moderator: Vanessa Fox, Contributing Editor, Search Engine Land

Q&A Moderator: Alex Bennert, In House SEO, Wall Street Journal


John Andrews, , Seattle SEO Consultant
Jordan LeBaron, Senior Consultant, Omniture, An Adobe Company
Branko Rihtman, R&D SEO Specialist, Whiteweb
Conrad Saam, Marketing, Avvo

And here we go!

Vanessa’s up at the podium, starting things off, going over the usual introductions, but in Vanessa’s fun way 🙂


First up is Conrad.

You hear the word “statistically relevant” – what does that mean?  If we were to test ten people in this room, and the men were all 4′ taller than the women, you’d be confident that they’re all taller.  If they’re all 4″ taller, you’d be less confident that they would be taller.  We need to look at how we come up with averages.  How confident am I in my decision?

Sometimes you don’t need to test – a major change can be seen as to its impact and it’s really obvious.

Fundamentals of testing include


Sampling Size


Confidence Interval

Types of test include:

Continuous Tests

Binary Tests


With sampling, you take some set of data and extrapolate that sampling to the population.  If you have a large data sent, you have less variability than if your data set is much smaller.

Conrad is using bell curve slides to talk about sample sizes and confidence levels. There’s no way I’m going to explain them here.

And now it went from bad to worse.  He’s talking about using something called a “TTest” (yes, that’s correct, two “T”s in that so it sounds like  “T-Test”.  Not to be confused with the process of testing green tea vs. black tea.  We’re talking about complex functions and formulas here people.  Way over my pay scale.  And geek-quotient.

According to, what it boils down to is that you need to find out ways to improve the quality of your data by increasing the sample size of your test.  And now I’m thinking – uh, why couldn’t you have just said that in the first place?  But then probably half the attendees here are just eating this stuff up because they’re statistical fanatics. 🙂

He says you can use an A/B Testing Confidence Calculator from, which will give you a confidence level assessment based on your test sample size.  And THAT is useful, even to me!  Thanks Dude!

Conrad says there are “tons” of this type of calculator out there as well. (just Google it and you see a lot of opitons)

Seasonal variability – nobody looks for laywers during thanksgiving, so if you test for people who search for lawyers and you do so during Thanksgiving, you have a bad data set.

Pay attention to things that might cause your sampling or your data set to be incorrect due to mistakes or assumptions that aren’t valid!


Next up is John!

I’m going to talk about the community aspect of testing.  As an SEO I want actionable information, I want to isolate myself from sudden changes, and I want to avoid penalties and protect against competition.

These days we’re hearing about better correlations, higher probabilities, super LARGE numbers like 11 million…

Scientific reports are usually supporting claims, like Page Rank Sculpting “Doesn’t Work” or Title tags should be 165 characters, only the first link on a page matters…

The value has shifted from the data to the claim, but that’s marketing, not science.

Scientists go through a peer review process before publication.  As Scientists, you usually can’t “redo” your study or republish your paper.  Promotion / fame is based on citations, respect, publishers protect their own quality score.  All of this is built into the scientific world.

In SEO, we have almost the opposite.  Remarkable claims get the most attention.  Sponsors fund studies.  There’s virtually no peer review.  Success follows attention, not validity, anyone can publish anything on the web, links are cheap and easy.

Instead of saying what’s true, share what you have and let’s ALL determine what’s true.

Remarkable claims backed by weak science draws lots of links, but they tend to be lower quality over time.

Contribute to the science of SEO

Science is slow, boring, not easy, and it’s expensive.  Most scientific experiments don’t produce significant results.  Scientists learn by making mistakes, proving themselves wrong, “We”, never make mistakes.  But every SEO is an experimentalist.

Publish your findings without making any claims.  Describe “What I Did”, “What I Saw”.  Be complete and transparent.  Let your peer community review your data.  Even if it’s not public, give peers access.  They might repeat your experiment.  They’ll provide a citation to your data if they do a write up on it as well.  This linking is very powerful linking.

Make suggestions about the data, but not “hard claims”.  You’re only making honest observation so nobody can dispute it.  Others are likely to promote hard claims on your behalf, as a one-off.

You can earn more valuable inbound links by publishing your observations and separately publish the discussions.  You’ll get more authority sites linking to your data.

You won’t get the social media attention because it’s not wild claims.  but the higher authority links are worth it.


Next up is Jordan

How do you kill a vampire? What are the two primary ways?  Audience answers stake through the heart, or direct sunlight.  These are the historic beliefs.  More recently, an expert has said that’s wrong.  (He flashes a “Twilight” book cover on the screen. )

How do we deal with this discrepancy?  Do testing yourself.

Now there’s a photo of Matt Cutts up on the screen and he says “Don’t Trust This Guy”.  And a photoshopped photo of Matt’s there now, looking really evil.  🙂

According to, you need to take what others say with a grain of salt.  Every site is different, every competitor situation is different.  You need to test for yourself.

Why do we test?  To help break through the red tape that’s out there, and validate the recommendations and processes we want to go through.  We need the evidence to show what we want to do is the right thing to do.

Testing process:

  • Formulate your plan – lay out the steps.
  • Execute your test
  • Monitor the metrics and reports you determined during the plan
  • Share the information.  there are key stakeholders in your organization that need to know the information.  If you can, share it online in the SEO community.
  • Maintain consistency for ongoing success.

Measure impact and conversions – use A/B testing with changes on your site and monitor the conversions.  Was there a negative impact on conversions?  If not, you’re safe to move ahead.

Data points to consider:

  • Visits or Searches
  • Average SERPs (pull data from visitor URL)
  • KPIs – your key success metrics

Generate trend reports, and granular data reports.

Example reports – Overall visits, Organic visits, Google Visits, Keyword group level results, all the way down to monitoring individual keyword results over time.  At the deepest level, you can look at results over time for an individual page.

Create a dashboard – to automate the generation of the data, which frees you up to do other work.


And our last speaker is Branko (Go @Neyne!)

I admire Branko – an SEO scientist who routinely does lab testing of the biological kind!

Some of the things important in science that can be applied to SEO:

First, we want to define the question, usually after we observe something.  The phrase isn’t “Eureka”, it’s usually “oh – that’s funny” – a thought that comes up as we’re observing something.  You don’t have time or the ability to test everything – its important to know what we can test or we can’t test.  PageRank is an example – they can test that in MountainView, but we can’t.  Everything you do on your web site is a potential thing we can test.

Gather information and resources before you test – related blog posts, social media, forums – maybe someone has done this before that you can learn from.  A problem with this is non-standardized terminology – we don’t all use the same words to describe things we test.

Perform an experiment and collect data.  Testing with “nonsensical” terms (non-words) is not really valid.  On the other end, testing for “payday loans” isn’t valid either.

I like to take the third route – phrases that are made up of real words, but aren’t normally phrases.  You’re able to couple changes to your site with changes in rankings.

Multi-directional experiments -I take a site without a link and change it, then look at the results, I’ll then do the reverse – if I see similar results, I have more confidence in the results.

Interpret your data and draw conclusions – does my conclusion agree with expectations?  Does it have an alternative explanation?  Bounce your findings off of others – two heads are better than one.

Remember – there are no definite conclusions.  That should be kept in mind.  It’s a tough reality to live with, but it’s important to be aware of.

Data Analysis- Get a real statistician to look at your data because it can lead to bad examples if you rely on yourself or someone who isn’t an expert.

Avoid personal bias.  We think we see what we do based on our expectations.  Try to erase your expectations before you look at the data.

Go Social!  Some of the best findings I’ve found have come from going social – you’ll learn much more.

SEO Testing Secret Ingredient:

Identify the people who like experiments, buy them a beer – you’ll get much better results and we’ll all enjoy it more!


Question time!

When you’re doing testing on a moving target, what are the challenges?  The moving target is the algorithm.  How can you isolate that the changes you’ve made are what moves the data?
Branko – my multi-directional method does a pretty good job.  But the title of the whole thing, you’re only increasing the certainty a little bit.

Vanessa – I really liked not only the multi-directional method but doing the test a couple times.

Branko – On a few pages as well.

John – Over time, you find some sites that are representative of content strategies – and you can go to them and see what happened to them.

Vanessa – if you’re looking at all the sites you’re involved with, it’s more likely to not be a coincidence.

Question: Do you want to talk about tests you’ve done that didn’t match expectations?

Branko – A while back it wasn’t well known at the time, but we discovered that you didn’t need quality links, only consistently more links coming in.  That was new to us.  It may not be that way today but it was new and it worked for us.

Question: Do you use external control groups? Do you track other peoples web sites to see if their changes are comparable to yours?

Conrad – Benchmarking competitors can be bad.  We do watch competitors like a hawk, but if you’re studying them, you might stumble into a test, assume that’s something they’re doing not just as a test.

Branko – I do it sometimes when I test links.  I insert a link to a control group that I don’t have control over – if that link behaves as links to my web site, then it’s better ( for my evaluation of my testing).

Question – where are good places people can go to publish this information, and exchange ideas?

John – I joined Twitter only for SEO and over time, I can twitter something and get attention of hundreds of people.  That same thing happens at forums – I like the SEOBook forum – they’re hardcore people with being serious.  In the science world, we form working groups all the time.  When you meet people with common areas, take advantage of that.

Branko – a study that came out on the Distilled blog – they published excels with data that anybody could take and study and we joined forces – and that’s the whole point of that.

Question: Can you each briefly give ideas for setting up SEO testing environments?

John – If you’re beginning, go into Google – Analytics, Webmaster Tools – put them to use

Branko – The problem with testing – it’s isolated from the places we want to be at.  I would make sure you’re constantly aware of what’s happening on YOUR web site – or isolating sections on your site and doing experiments – that teaches you more.

Jordan – if you can’t set up a completely controlled environment – set aside testing areas of your site.

Conrad – Make sure you and the people you are sharing the data with are using statistical standards.

And that’s it!

About Alan Bleiweiss

Just another guy. Who happens to have a lot of experience living, breathing and sleeping organic SEO. So that's my primary focus - high end SEO audits and consulting for sites ranging from thousands to tens of millions of pages. In my spare time I blog, rant, write eBooks, and speak at industry conferences.

Read more from