A Look at Website Split Testing

We started split testing over at daveramsey.com several years ago. When we first started, we used custom code to pull it off. In 2008, we took the jump over to Omniture’s Test and Target product line.

A/B Split Testing is an important tool in the arsenal of online business. That’s one of the reasons I found this post by 37 Signals so interesting. It was extremely clear and straight to the point.

37 Signals - Baseline Test

37 Signals - A/B Split Testing

Contrasting the Adobe/Omniture vs. Amazon/Zappos Acquisitions

First off, let me say this is not going to be some super in depth analysis that looks into financial statements, revenues shifts, profitability or details like that. I’m merely going to make some basic observations and speak to how I as an Adobe/Omniture customer feel. First, some background.

A Little Background
Our company as been an Omniture client since summer 2007. We’ve been a heavy Adobe customer since 2005 (meaning we have 20 creatives now), and before that we were a heavy Macromedia customer. We use their ColdFusion Enterprise servers…several of them since our site handles 1,000,000+ uniques a month. We use lots of Photoshop, some Illustrator, some Dreamweaver, quite a bit of After Effects, some Flex and probably some things I don’t know about.

Since adopting Omniture as our analytics platform in 2007, we’ve been gung ho pushing analytics into everthing we do. We opted for the full time analyst right out of the gate. After 3 years of sculpting, he’s a rock star. We started with Omniture’s Site Catalyst and Search Center. We’re now using Omniture’s…I mean Adobe’s Test and Target powered by Omniture as well as Omniture’s…I mean Adobe’s Discover 2 powered by Omniture. We love the tools and we’ve done some great things with them. The main reason for us moving to their platform was in preparation for the launch of our new website…which I might add went off quite successfully in November 2009…thanks to Omniture’s…now Adobe’s Tools powered by Omniture.

Here’s Where I’m Weirded Out
I may have been in a hole or something since I stay so busy, but this week I started noticing that my beloved Omniture Green is well…gone. Where did it go? I can’t find it anywhere. It’s like we’re entering fall and all the leaves are falling off the trees! It’s disappeared from my regular Omniture (now Adobe) emails. It’s disappeared from Omniture.com. It’s disappearing from the SiteCatalyst tools!

Now honestly one might think. Well, duh. They were just acquired by Adobe. Of course the branding will change. Just being candid, my day is so busy I hadn’t really thought about it much. Now that it’s taking place before me, I’m having some weird attachment emotions. What is this I’m feeling? Well, actually there’s a good answer. Several incredible marketers have written about it. You might know some of them. Seth Godin, Martin Lindstrom, Patrick Hanlon just to name a few. So it’s kind of funny to hear these guys speak for years and read their books then see what they teach actually affect your emotions about a brand you deal with all the time.

Time for a Contrast
The Adobe/Omniture merger went through in late 2009, about 10 months ago. Another big merger happened recently. Amazon and Zappos. Let’s look at where they are after 1 year…roughly the same time frame.

Zappos before Amazon Merger
Zappos before Amazon Merger
Zappos After Amazon Merger
Zappos After Amazon Merger
Omniture Before Adobe Merger
Omniture Before Adobe Merger
Omniture After Adobe Merger
Omniture After Adobe Merger

My only point is I’m basically an Omniture freak and why not. Omniture’s helped move our company forward, move our mission forward. Since 2007 when we moved to Omniture from Google Analytics, our web staff has grown from 35 to 60. Omniture helped us do that. Our revs grew…a lot. We just launched a new website that accounts for 75-80% of our company’s business all based on what Omniture helped us do. I’m a bit touchy when you go messing with one of my favorite brands, it’s colors and it’s employees.

Maybe it’s sophisticated rebranding, but it seems like a miscue when you already have the most successful web analytics brand on the market. Brands are emotional as Martin Lindstrom showed through Buyology, Patrick Hanlon covers in Primal Branding and Seth can’t stop talking about it.  You took away my green…you just gutted part of my emotional attachment to the Omniture Brand. Not a good way to finish out the first year. I know it sounds a little weird, but it’s where I’m at.

Adobe, I hope you get this right. 😉

2010 Omniture Summit – Day 1

Like many Omniture attendees, my day started around 4:30. It was a little earlier than I had planned since our baby was up, and my loving wife felt compelled to wake me as well. It was just 30 minutes early, so I survived. After that, it was off to the airport for the flight to Salt Lake City. The flight was good and the conversation was better. Being at the office working sometimes doesn’t allow enough time for real discussion. It was good to have several hours of business chat with Jen Sievertsen our Executive Director of Internet Marketing.

Arriving at 11:30, I was pressed for time to get over to the hotel and prep for this year’s Omniture MindMeld session. I enjoyed the MindMeld run by Matt Langie last year, and was pleased to participate again this year. Matt did a great job of making some changes for this years MindMeld and found a way to improve on an already good event. The breakout sessions remained in tact, but the closing discussion format changed from a six single person 10 minute presentations to a joint 20 minute presentation with questions from the audience. It seemed everyone enjoyed it and found some value in it.

At the MindMeld, I was fortunate to be able to lead a discussion table on The Integration of Data: Bringing Online and Offline Together. This is something that’s in the future for our organization. I see this as a huge opportunity spot for us. During the pre-session, I was able to chat wtih Michael Thomas, Director of E-Marketing for E*Trade Financial, Jane Kell, Manager of Web Analytics for Delta Airlines, Joel Wright of Dell’s eIntelligence group, and Alex Lanshur, President and Founder of PublicInsite.

After the MindMeld session was over, I headed over to the mixer where I got to spend some time with Joe Megibow, VP of Global Analytics and Optimization for Expedia.com. Joe is a great guy with an INCREDIBLE mind for analytics. I also spent some time talking with Brian Katz of VKI Studios. Brian was also at the table discussion I led and he had a lot of useful, quality information.

Then it was off to check on our presentation being ready for tomorrow and down to the dining area. As usual, the food was good, the networking was great, and a little music from the live band added a lot of energy.

I hope to start posting some flip videos of the attendees tomorrow so stay tuned!

Time for bed, day two starts in 8 hours.

Rachel and Jane at MindMeld
Rachel Scotto of Scotto Consulting and Jane Kell of Delta Airlines
Michael Thomas of Etrade
Michael Thomas of Etrade
MindMeld Attendees
MindMeld Attendees
Alex Langshur, Rachel Scotto, Matt Langie and Me
Alex Langshur, Rachel Scotto, Matt Langie and Me
The Band at the Opening Reception
The Band at the Opening Reception
Shot of the Opening Reception
Shot of the Opening Reception

Omniture Summit 2010

Since our company’s relationship with Omniture began back in 2007, we’ve steadily been increasing our expertise with their product line. While I’m sure there are many more power users out there who exceed our current level, I feel our efforts have positioned us nicely in the Omniture community. Being positioned as a top 3,000 US website, depending on the day, we face a lot of challenges with our site. DaveRamsey.com supports some 25 different businesses covering several flavors of B2C commerce, B2B commerce, and about 20 different lead types for the various businesses while trying to provide free content and tools for our visitors. Needless to say, it is very challenging.

Omniture has been a fabulous tool for us as we’ve attacked our online design and business problems to increase results. Through our success with Omniture, we’ve been able to conduct on online interview with John Broady, Executive Director at OTTO Digital, contribute to an Omniture case study covering daveramsey.com, participate in the first MindMeld, and most recently contributed to an Omniture Webinar on The Cost of Free.

We’ve recently been pleased to speak with Matt Langie, about the possibility of being a presenter at the 2010 Omniture Summit. This will be our 4th Summit to attend as a company, and we are looking forward to it. Omniture has been a key component to our online strategy over the last 4 years, and we are excited about the opportunity to share some of what we’ve learned with the industry at large.

Omniture Webinar – The Cost of Free

Tim Munsell, lead web analyst for daveramsey.com, and I were glad to participate in a webinar (list of all Omniture webinars) for Omniture this week. As a result of Tim and the web team’s work on daveramsey.com, we’ve been able to accomplish some pretty remarkable things using Omniture’s tools. The webinar was a great experience for us and we appreciated the opportunity. Thanks Matt Langie and Mike Barton.

Tim has done a great job for us since taking on the title of Lead Web Analyst back in 2007. Since then, we’ve added both Omniture’s Test and Target tool and their Discover tool. All of these tools have been a tremendous help for our web initiatives and keeping our annual growth on track.

Omniture Launches CMO.com

While I was at the Omniture Summit back in February, their executive team announced the upcoming CMO site. Omniture finally launched the site, cmo.com, on August 27th. If you are into marketing or shoulder any of the business responsibility for your organization, Omniture’s CMO site is something you should spend some time checking out.

The site is designed in Omniture green, but other than that, Omniture has done a very admirable job not saturating the website their marketing. Just another example of Omniture’s classy approach and drive to deliver excellent products. I poked around the site just a little tonight to see if I could find some interesting features to mention.

Why CMO.com?

Today, a new breed of marketing executive is emerging to transform companies, overcome historically complex marketing challenges, and realize the array of new opportunities our increasingly digital world offers.

These new executives are Digital CMOs.  These are CMOs who have learned to leverage the power of data to understand and segment their customers.  They harness the power of digital media to drive precision marketing and customer engagement.  Finally, they utilize technology to measure how well their programs perform against key business objectives.  In short, these CMOs use digital marketing to deliver. (cont’d at cmo.com)

CMO Interviews
Robert Turtledove – For Robert Turtledove, H&R Block’s recently appointed chief marketing officer, it’s tax time year-round. “The wiser you spend, the more bang you get for your buck,” he says in this interview with AdWeek.

CMO Surveys
Teens and Mobile Phones Over the Past Five Years: Pew Internet Looks Back
Publication: Pew Internet and America Life Project
Teens have previously lagged behind adults in their ownership of cell phones, but several years of survey data collected by Pew Internet show that those ages 12-17 are closing the gap.

Insight Section
An insight section has also been included that covers just about every topic of marketing you can imagine. The insight categories have been nicely broken down for easy scanning and focus: Online Media, Traditional Media, Emerging Media, Tools, Branding and Communications, Lifecycle Marketing, Market Research, Analysis and Measurement, Strategic Planning and General Management.

After a quick look around, cmo.com seems to be off to a good start. I look forward to seeing how this new venture for Omniture plays out.

Omniture Press Release featuring DaveRamsey.com

I was excited to see that some hard work finally paid off. It was 2 years ago that we decided to use Omniture as our web analytics provider. By July 2007, we had completed our implementation and had started leveraging analytics to improve our website. We’ve been very pleased with Omniture’s tools and have seen some very good results.

One of the challenges and goals our web team has set is to become influencers of the web and internet technology. While we are relatively small business at 260 employees with 55 web staff members, we have been able to make some headway into getting noticed by other web professionals and businesses.

Some of the highlights for 2008 included

  • Mapquest mapping technology. One of the first companies to leverage their new API toolset (visit post on webmonkeyswithlaserbeams.wordpress.com
    Post 1
    Post 2
  • Automated streaming and podcast posting system. Streaming archives are live on our site 15 minutes within of The Dave Ramsey Show ending. Podcasts are available within 1 hour. The entire system is automated which includes cutting out the commercials.
  • Unique usage of Omniture marketing tools for geo-targeting. We implemented what we call geo-exclusion marketing to present relevant ads to state specific markets.

In 2008, we were able to step up our involvement with Omniture by conducting an interview with John Broady which he later posted on his blog. This interview covered our implementation and results of leveragin Omniture’s Test and Target marketing tool. In February 2009, I was invited to the Omniture Summit MindMeld. The MindMeld was a great opportunity to network with some of the industries leaders like Sony, Orbitz, Time Warner, Comcast, and a few others.

As a continuation of our Omniture efforts, Omniture put out a press release today which was picked up by Yahoo. View the press release.

Omniture Summit MindMeld Session Videos

The videos from the Omniture Summit MindMeld Sessions were posted today.

Some links related to the MindMeld
Omniture Summit MindMeld Press Release

MindMeld participant comments – Stephane Hamel

My previous Omniture Summit MindMeld post

The MindMeld videos

Stephane Hamel
Tony Bradshaw

Gary Angel
Bill Gassman

The video turned out better than I thought it would be. It is the first time I’ve seen myself present on camera, and I’m looking forward to practicing in front of a camera more. It’s kind of fun and VERY helpful in developing your presentation style. Give me your feedback on my video. Thanks!

Omniture Summit 2009 – Thoughts and Summary

Overall, I was very pleased with the Omniture Summit. Our company sent 3 people to the event.

  1. Me, VP of Internet Business and Technology
  2. Tim Munsell, Web Analyst
  3. Branden O’neil, Web Marketing and Test and Target

Just a quick tidbit about our team to provide a basis on my thoughts about the event. We have 55 web staff in our company. 50 of those report to me and consist of 30 programmers/technology, 11 marketers, 5 creatives and some admin people. Our web group combines the tech and marketing into one group and it works well for us.

Now for my take on the event.

1) Strategy and application
The Omniture Summit 2009 had a good balance of thought provoking keynotes for the strategists (like me) in attendence while also providing some good practical application for the analysts. The keynotes weighed more heavily with strategist related content while the breakout sessions were more focused on the analysts and actually using Omniture’s products to accomplish business goals.

Since I’m both a strategist and love numbers, I get a lot out of the keynotes as well as the breakout sessions. However, in some of the conversations with other attendees, I came to realize that the audience is largely composed of analysts a.k.a. “get it done/pull the numbers” types. One Sr. Analysts who leads a team mentioned that several of the Keynotes weren’t practical or needed for him or his team.

In planning for Omniture Summit 2010, it might be worthwhile to do try something…if it hasn’t been done yet. 😉 Survey  of the Omniture attendees of the 2009 event and let them “build their own event” via survey information for the 2010 event. If this is feasible, it could easily start via an online survey in a follow-up email to the event. “We hoped you enjoyed this year’s Omniture Summit. We value our customers…etc…etc. Please take a moment to provide us with feedback on the Summit and how we can make it even more valuable for  you in 2010.” Then provide a short survey followed by a long survey or serve them simultaneously. Perhaps approach it by sending the short form survey to collect some basic info. Then for those that complete the short form, ask them if they would also complete the long survey at their convenience.

Just my 2 cents.

The sessions were good, but for the newbie and experienced alike, it can be a little difficult to figure out which sessions you should attend. Honestly, its the same for any conference. I think the tracks work well in the agenda, but there may be some room for improvement. I’d love to see Omniture innovate in this area. I don’t have it figured out, but here are a couple of suggestions.

  1. On the Summit Agenda, add some more qualifiers for each session. Type 1 Difficulty – Basic, Intermediate, Advanced. Type 2 Appeals to – Implementers, Analyzers, Strategists
  2. The session format works and generally seems to follow (a) highlight problem or opportunity, (b) familiarize audience with Omniture product, (c) user case study. I personally like the format. Some of the presentations come off better than others which is typical at any conference. I’ve never walked out of an Omniture session due to it being lame or presented horrendously, however, I have walked out of sessions at virtually every other conference I’ve attended.

    A couple of suggestions, perhaps 2 screens could be utilized per session. 1 for current slide, 1 for the last slide. I frequently found myself trying to keep up with most of the speakers and missing some key points.Curiosity item. How scrutinized are the presentations and at what level? With most companies, workload impacts how heavily a presentation is reviewed and by whom. Are the best people looking over the material for quality and flow?

I was able to get something valuable out of all the sessions and keynotes I attended. I did jump from 1 session to a 2nd not finishing either.

2) Omniture Service and Accessibility
Omniture is second to none in serving their customers. That’s evident during the Summit at every turn. After being familiar with the company for 3 years, I honestly believe their drive for service excellence is only getting better. Of course it suffers occassionally due to growth, but they’ve adapted each time and maintained a high level of excellence.

I actually bumped in the one of their Sr. Execs during the last session on Thursday. John Mellor, EVP, Business Development and Corporate Strategy. Have you ever had one of those moments when you look at someone and know they look familiar, but you just can’t place them? It’s embarrasing. All I recall is walking over to John to say “hi” probably followed by, “you look familiar. Did we meet at the Summit last year?” The shock on John’s face…well…wasn’t priceless, but it did get an apology from me. 🙂 John was way too gracious in that moment. He politely struck up a conversation with me then extended his hand and a business card.

This is typical of all the Omniture Execs I’ve met. They are very personable and approachable. They genuinely care about their customers and want to be part of their success.

Another great example of this was at a Wednesday night mixer party. There were about 50 people in attendance with some coming and going each hour. Branden, one of our guys, was there and spent some time talking with Josh James. Now I don’t know about you, but I understand the typical M.O. of a $300 million CEO talking to a Mid-level web marketer who’s been using one of your products for less than 6 months to be “Don’t bother me. I don’t have time for you.” Josh and his executive team’s “serve the customer” attitude permeates the entire organization…at least all of the people we’ve dealt with. Omniture gets high marks from me in this area.

3) Sponsors Showcase
I didn’t spend as much time in the sponsors showcase this year, but I did inquire on 2 services that we’ve been interested in. Responsys and Bazaar Voice. The sponsors showcase was a quality deal. Not too showy, but very practical. In talking with the vendors, you can tell that the sponsors are tightly integrated in the Omniture system/culture. It wasn’t your typical tech event where you feel like meat and the vendors are doing everything possible to get your attention. Very low pressure at the Summit. Very informative, serving the attendees and useful.

4) The Food
What conference can you go to that provides not only knowledge, but also the food? Food was in abundance and variety. While this may not be the reason you come to the conference, it sure does help offset the budget when you make the pitch to get approval to attend. 3 guys x 3 days x $50 = $450. Not to mention you get to stay focused on learning rather than where to get your next meal. Omniture covers the food in style. Unfortunately, I gained 5 lbs.

5) Community
One of the biggest values of attending the Omniture Summit is meeting other professionals. If you’re a people person and an Omniture customer, there’s no better place to be. You’ll make tons of relationships that you can learn from and culture for years to come. Take Karl Rainhold from Nike for instance, this guy must be one of the gurus of the Omniture system. He’s been to at least 7 Summits. I met him 2 years ago in passing. I also have been fortunate to meet Paul Strupp of Sun Microsystems (2007 Interview). Paul has turned out to be a perennial Summit associate for me. He’s a die hard Omniture user and expert. I love seeing him at each Summit and covering the bases. My hope is that one day, I’ll be able to contribute some value for him on his projects. He’ll probably kill me for posting this link, but someone may find it useful. Basics of Analytics – 2005

Ending
I give the 2009 Omniture Summit high marks. Omniture is a solid company with a good vision, great employees, and a industry impacting culture. I’ll be a customer for years to come.

Omniture Summit 2009 – Ski Day

The conference ended Thursday night and I must admit that I’m a bit tired. No not due to the parties. I’ve been up late blogging about the event. I started blogging about the event on Tuesday Morning prior to the Mind Meld session. After running into Brian Watkins, Omniture’s Sr. PR manager over social media…fondly referred to as Chief Canary Officer for his affinity with Twitter, I came to the realization that I should cover the entire event. Finish what you start right?

For the die hards who can figure out some way to stay in Salt Lake City 1 more day, Omniture provides free lift tickets and ski rental discounts along with a $20 gift card for the Snowbird Resort. I  used some of my built up vacation time to enjoy the fresh powder.

The 2008 Summit was my first ever run at skiing, and it wasn’t pretty. I did learn some things though. This year, I was eager to try a snowboard. I’ve developed some proficiency with skateboarding and wakeboarding, so snowboarding seemed to be a natural progression. We had some errands to run this morning, and then finally got on the slopes around 11:00. Beth, Tim and I stayed on green trails today while our comrade Branden ran all the blue trails and a black diamond. We wrapped up around 5:00 and headed back into town. Thankfully, no broken bones or injuries, but we’ll have some sore spots tomorrow. Skiing was a wonderful way to close out the week.

Thanks Omniture for providing some time to relax at the end of a hard week.