Friday, January 27, 2006

Paid Search Testing: A New Study From Jim Novo

Page views, Conversions & ROI from the perspective of THE MARKETER!?

If you've ever wondered what types of paid search tests you could be running with clients offering a bountiful budget, check out this study by Jim Novo from the Yahoo Analytics Group Discussion. I've postd the entire thread for you to follow. It's a great sutyd and it addresses a lot of concerns we have as search marketers highlighting the suspicions we all have when trying to associate the measurement of our paid campaigns to the overall successes of the the campaign as a whole. thanks Jim!

> I guess what I'm saying is that the only results from a PPC campaign

> that matter are the *incremental* clicks that paid search provides

> above and beyond organic search. And if those incremental results

> aren't significantly higher, it's a lot harder for me to justify the

> spend... 'ego spending' of course aside.

Exactly, and the question if you are optimizing *profits* as opposed to

sales or "exposure" is where is breakeven on the PPC? This of course

depends on the margin of the business and the cost of the click, but there

are several other dynamics in play, including the following:

1. The tendency of a 1st ranking PPC to deliver sub-optimal ROI due to

"sport-clicking" by causal / newbie "surfers"

2. The tendency of lower-ranking PPC to deliver higher ROI due to bid gaps

and (often) dramatically lower costs

3. The tendency of "deep searchers" - people who click on lower ranking

organic and PPC links - to be further into the research cycle / more likely

to convert to final objective

So, for example, in one 2003 test, a page ranked organically as #2 for a

certain high-volume phrase. This same page content was used to create a

landing page for a PPC campaign using the same search phrase. The test was

conducted in March, neutral seasonality for this business.

With a #1 PPC ranking, the PPC campaign generated 11% in *incremental*

sales with a 60-day latency tail but had a negative 12% ROI on margin minus

overhead (note specific use of the word incremental, which I will explain

further below). When this same listing was dropped down to "deep shoppers"

at PPC rank #4 (in this case, 1st ad at bottom of Yahoo page), it generated

4% incremental sales with a 1786% ROI on margin minus overhead for the same

60-day latency tail. That's almost 18 : 1 payout. Without the tail

(first conversion only), it was 623% (almost 7 :1).

In addition, the "deep shopper" segment on average had a 70% repeat

purchase rate as opposed to 58% for the #1 PPC position. So even the "tail

of the tail" was better at position 4. This was on the higest volume

search phrase for the site, so it made a huge impact on overall


Remember, the landing pages for both the high ranking Organic link and the

#1 ranking PPC link were exactly the same - layout, copy, all of it.

Now, the reason I specifically used the word incremental is we had control,

which demonstrates a real dark side of PPC with Top 3 ranking Organic.

When the #1 ranking PPC ran with the #2 Organic link, the sales *volume*

coming from the PPC link ran about 43% versus 57% for the organic link.

This ties pretty closely with some recent studies on click behavior (on

average, 60% click organic, 40% click paid).

But dig what this really means: if incremental sales are 11% versus control

(no PPC) and sales volume with PPC is 43% PPC, that means nearly 77% of PPC

sales were **stolen from the organic side** - they would have happened

anyway without the PPC link.

Factor this "media cannibalization" into ROI, and now we're down around

negative 48% ROI for the test #1 ranking PPC with organic at #2. For every

$1 we spend we lose 48 cents - 12 cents in tangible ROI, and 36 cents in

"Opportunity ROI" - ROI we won't get because we wasted the click budget by

not using it to buy "real" incremental clicks.

Hey, just increase the budget, we'll make it up on volume!

This is precisely the situation I was alluding to in the first post on this


The visitor clicked PPC, landed, went Back, found Organic, converted. With

all this occuring in the same session, it's highly likely that paid click

is a pure subsidy cost - conversion would have happened anyway. To peg

this click as "source" gives credit where credit is (probably) not due, and

leads to eroding margins through increased PPC subsidy costs.

> So back to the original question, can you think of a solid way this

> incrementality could be tested?

"Solid" way? Well, I guess that would depend on the technology that was

available and the sales volume you are talking about. Not sure you can

truly "A / B / C" it without some significant bid management / search

engine API technology. I'm not sure the engines would give up that kind of

control - though I bet THEY have tested it at some level.

The above test was a 3 week manual "alternating days" test on Overture for

a store with 500K - $1 million annual sales and average order size about

$72, so there wasn't a lot of room for high tech tools.

If you run A on Monday, B on Tuesday, control / C on Wednesday, then start

over with A on Thursday and continue this rotation, the following Monday B

will run and on Tuesday C will run etc., so by the end of 3 weeks you will

have A, B, and Control data normalized by day - both campaigns and control

ran on every day of the week. Not a statistically pure methodology, but

not horrible, for sure - and cheap! As Robin said, "I'd rather do it

unscientifically (e.g. pull the ads for a time period that is longer than

the average latency and measure) and swallow the error rather than just

"attribute" the conversion."

If the result spread is significant enough, as it was on this test, I'll

give up points in accuracy to get closer to the "directional truth". Each

of the top 30 search phrases *where there was a top 3 organic ranking for

the phrase* was optimized in this way with the results very directionally

consistent across all phrases. It was almost always more profitable to

have a lower than #1 paid ranking when a top 3 organic ranking was present.

Below the Top 30 phrases, some of the lower volume phrases produced

inconsistent results which was probably a result of test methodology error

/ lack of frequency.

While it may not be "practical" for large scale retailers to test like

this, you would think for certain high volume phrases it would be worth

poking around it a bit given the potential for cost savings. Definitely

not worth thinking about if sales is the focus, because (probably

unprofitable) sales will be lost without the #1 PPC listing.

Are there any analytics providers with a direct interface to the search

engines capable of generating / controlling / measuring this kind of

testing on a large scale? Any analytics providers that interface with

complex bid management systems like Kevin Lee's ? I don't know

the answer.

Of course, changes in the way paid listings are displayed (often related to

how many bidders there are) can change the outcome of this test. The

results on Google were also directionally consistent though less dramatic.

I assume this is because of the different PPC display approach versus

Yahoo, but perhaps also due to some of the "alchemy" surrounding Google PPC

rankings, which are more difficult to control on Google.

And for sure, there are reasons people buy PPC other than to drive profits,

so being #1 may be worth it, but the true costs should be quantified.


Wednesday, January 04, 2006

Do-It-Yourself SEO: Now Uncle Joe Can Optimize Too

Seriously now. If you are going to advertise the ultimate SEO software solution, you should really be ranking pretty high for terms like "seo software." Sites like just give a bad name for those of us who work in this industry and have busted our butts to get where we are today. It's not just about TITLE and META tags, writing a ton of content and trading links. Many of us come from traditional backgrounds spawning from the web like designers, developers, producers, copywriters, etc. I can speak for myself that we not only have an understanding of how to grow a site from an idea to a sketch to a mockup to slicing it up to programming it to...the list goes on, we are experts in this field of search now. Most local business owners think "algorithms" is greek. Hiring a firm, agency or even freelancer to manage your search marketing efforts is a cost-effective way to make sure the rest of the things in your business are running smoothly and being tended too like accounting, finances, affiliate programs, offline marketing, media buying/planning and making sure the carpet isn't coming up. A do-it-yourself solution isn't wise money spent unless you are VERY familair with this field, attended some webinars or trade shows or took a course at At some point you truly need an expert to evaluate what you are doing. It makes sense and unless you have a money in the back of your business; be sensible and make sure your dollars are earning you something valuable in return.