You Probably Don't Need A/B Testing

welcome to the ecommerce podcast with matt edmondson a show that brings you regular
interviews tips and tools for building your business online [Music]
hi and welcome to the ecommerce podcast with me your host matt edmondson all of this week's notes and links can be found
on our website at ecommercepodcast.net forward slash that's right it's episode
i think that deserves a big cheer and maybe the audio guys can add that in
now a b testing is a popular technique that a lot of people think is essential
for online success but in today's podcast you'll see you probably don't need it no i
appreciate that's a little bit controversial and if you are like most people you've probably heard of it
and think actually it is essential for online success but today's guest is here to tell us that
that's not always the case and in fact in many cases you suck you can succeed
without ever using a b testing so stick around and let today's guest a chap
called oliver palmer explain his theory in much more detail you are not going to
want to go anywhere hey there are you a business owner here at oregon digital we know first hand
that running an ecommerce business can be really hard work as the online space gets more
competitive it is becoming even more challenging to stay ahead of the curve we totally get it so we want to help you
succeed by offering a wide range of services from fulfillment marketing customer service and even coaching and
consulting just so that you can do what matters most save yourself the time and
the money and let us handle the day-to-day tasks this way you can run your business without having to worry
about the boring stuff so what do you say are we a good fit for each other come check us out at oreodigital.com and
let us know what you think
thanks for joining us on the ecommerce podcast it is great that you are here it
really is now whether you are just starting out on your ecommerce journey or if you're like me and many of the
others that listen to the show you've been around for a little while uh the goal of this show is simple we are here
to help you deliver e-commerce wow we want you to grow we want you to develop
we want you to conquer the digital sphere in which you operate and to help
you do that every week we bring you two things great show sponsors and great guests experts in their own fields with
stories with insights with principles that we can use and learn and adapt for our own
web stores yes we can now today's guest oliver palmer is an experimentation practitioner it's not an
easy thing to say it isn't but that's what he is and he's a conversion rate optimization
consultant he works with organizations to demystify experimentation and the martech stack
and get a better and or help us get a better understanding of how to integrate with the big organizational picture the
man is a legend he has got a lot to say about this topic i really really enjoyed
this conversation you're not going to want to miss it so grab your notebooks get your pens have your cup of tea ready
and without further ado here's my conversation with oliver so oliver thank
you so much for being on uh our podcast here on the other side of the world uh
almost really uh you're in melbourne i'm in liverpool but thanks to technology
we're having a conversation with each other yeah thanks a lot matt great to be here no nice brilliant it's brilliant so
we were we were if you're watching this podcast um uh on youtube then you'll notice from
the camera we were just joking about it weren't we the way the cameras are set up it looks like it's daytime for you and night time for me but in fact it is
a complete opposite it's first thing in the morning here and lasting at the night view just happens to be the way
the lighting works uh which is quite peculiar so it's night time for you uh in
what i assume is sunny australia let's live up to the stereotypes but it's night time for you
um what what what does the daytime normally hold for you sir
what does the daytime hold that's a funny question the daytime holds many things matt uh right now the
daytime holds going back to the office for the first time in uh in two years
how have you found that i've descended into outright hermitude i
think after two years tucked up in my little room here i'm sort of like do you have to be in my peripheral vision right
now making phone calls and things um but it's nice to be away from my
children [Laughter] that's so true though you kids younger
how old are you kids oh four and two okay yeah yeah you see my kids they're a
little bit older one of them's at uni uh my other son has just turned and my daughter's just about to turn so
uh i've not had you know i've to be fair my sheds down at the bottom of the garden so they tend not to bother me if
i'm honest with you side i didn't experience that when we're on lockdown but i you're not the first person i've
heard say it's great to get back to the office to escape the kids that's quite funny yeah i've been it's mixed blessings but after
two years yeah it's good
that's very true so you're back into the office um where you guys uh do conversion rate
optimization and yeah or cro as we like to call it i mean sort
of so my work is is as an independent consultant and lots of my work is
not so much being a cro practitioner anymore but really helping big organizations to get better at
experimentation so sort of making that leap from thinking that they have
all of their battle one expertise and know exactly what they should do and how they should proceed and never really
measuring the impact of their strategy and making the leap toward actually saying we don't know anything when we
measure what we think we know we're wrong and sort of taking this journey toward data-driven humility really which
is um harder than you might think of course that's an interesting phrase data-driven
humility i i i i've never heard that before but i
quite like it i've never said it before matt so it's a world premiere okay
you heard it here first ladies and gentlemen uh quick let's copyright that phrase uh let's just do this
that's brilliant so you work with um large organizations helping them realize
that they know nothing at all which to be fair is the job of any consultant in any field isn't it it's to help them
realize that actually um perhaps what they thought they knew and what they actually know are two different things um
have you found that because that that must be an interesting journey going from being a cro practitioner to being
a bit more headline consulting helping other people uh do what maybe you used to do
um yeah i mean it's been it's been interesting it's sort of it felt natural you know i was working as a practitioner
for about a decade really and so i felt like i'd sort of more or less done
everything i could do there and just realized that there was this sort of particular niche skill that i had which
is helping large organizations make that transition and i just sort of
came to realize i suppose that there was enough maturity in the market there were enough businesses that were willing to
make that leap um and that it it was a sort of a unique
skill set that i happen to have but for a long time i think it was a little bit too niche but we're sort of at a stage
now where most large organizations have been a b testing in one form or another
for you know five six seven eight nine ten years whether that's through an agency you know the marketing department
has an agency relationship or e-commerce works with an agency or something like that and and in a sort of in a
fragmented way you know they might have run a few experiments and then often what happens is you know
the cmo or the ceo or the managing director or someone goes to one of the big conferences goes to the adobe
conference in you know las vegas or wherever it is um sees all of the wonderful things that people are doing
with data and optimization and so on but typically buy is a you know very expensive tool from adobe or optimizely
or somebody else and then sets up a team and then probably six months later realizes that
they're not quite getting the results that they expected um so that's typically where i sort of fit in
that's really interesting again a great picture of somebody going to a conference and getting something that they probably shouldn't get at the
conference how many times does that help happen to me um so
one of the things that i remember from uh pre-call and one of the things that we should probably just jump into right
straight away is um this idea of a b testing now for the for those that are
maybe new to e-commerce um or who have never done any any sort of of this type of thing just quickly explain what you
mean by a b testing yeah so at its at its sort of simplest a
b testing is trying two different things simultaneously measuring the results and
seeing which one got the best result so it might be um the simplest thing like
changing the you know the the banner image on your home page for instance you know i know a lot of
retailers sort of agonize about the way that they merchandise their stores and what sort of promotions they choose to
put in different places so if you're having a you know a dispute
a friendly dispute around uh you know what should be on your home page for instance one of the ways that people
will typically settle those is to say well let's test it you know let's not just uh put the purple widgets let's try
the red widgets too and see if anyone does anything differently yeah that's a very very very
straightforward way of thinking about it and and if you've been around e-commerce for a little while you're going to have
heard either through instagram ads or through you know whatever magazine that you may
or may not read that actually you need to be split testing or a b testing um
and that is all part of optimization isn't it that actually this is this is in fact so many people now when
you hear of um cro or conversion rate optimization immediately think a b testing or split testing it's like okay
i now need to figure out if i change the color of the buy now button does it have any impact on sales yes or no
[Music] my experience though is you need quite a
it's not that straightforward if you're new to e-commerce and just starting up because you might not have the traffic
volume big enough to actually do a successful a b test ie can you get enough data to give you a meaningful
result or is it just an anomaly actually at that point in time we don't know so is there
is there like a have i got that right is there like a minimum effective dose when it comes to a b testing like a minimum
amount of traffic that we need to figure it all out i mean it's hard to say um
typically what you're dealing with is is um you know it's it's statistics it's
probability so in any a b test there is the the inputs which are essentially what is the number of uh visitors that i
have coming to the site across the duration of the experiment what is my baseline conversion rate for
instance and what is the minimum detectable effect so what is the what is
the minimum amount of uplift that i'm looking for so if you have um
let's say you have visitors to your home page every week for instance
um and you have a baseline conversion rate of three percent and the minimum detectable effect that
you're looking to achieve is one percent then it's going to take you
a lot longer than if you're looking for an uplift of or for instance
and when you sort of you put those calculations together and there's lots of a b testing if you google a b testing calculators there's lots of them out
there and you can see punch in the numbers on your own site and you'll see you've got to get an indication as to
how long you'll need to run an experiment for but really with those sorts of those
calculators what they tell you is the sweet spot typically is um in e-commerce it's very large retailers that have lots
of traffic and where a minimum detectable effect of one percent can mean millions of dollars
you know that's that's a real sweet spot and that's those are the sorts of clients that i typically work with um
whereas if you are much smaller and you you you know you have thousands of visitors to your site rather than
millions and really for it to make it worthwhile with all the effort that you put into building the test analyzing the
test and so on you really need a increase in conversion or revenue or something it becomes a lot uh
a lot less productive let's say yeah no that's fair enough i was always um
i remember having this conversation years ago with various people i was always intrigued by
um i don't know if it's called the one percent rule but it maybe it's just me that calls it that in
which case i should buy the onepercentral.com or something like that to go along with you know our
data-driven humility um but for me the one percent rule was such
that if i could find uh four or five things uh on my side that
gave me a one percent increase and actually the cumulative effect of that was quite significant during me and it
was i mean we that was sort of where our traffic was with our e-commerce sites that actually
um the cumulative effect of sort of four or five small changes that bought one
percent increases and we found that when we looked for those rather than trying to find the big things which were
let's face it you know almost impossible once you sort of got up and established
and that actually worked quite well as a strategy for us at that point in time i
don't know if you've got any thoughts on that whether whether actually you can create these sort of cumulative effects with
with these sort of smaller changes yeah i mean absolutely can i mean that's the sort of foundation on which a testing
program is built and you're quite right the you know the
you would only really expect to get them at the very beginning that's finding and fixing broken things that you weren't
previously aware of and then really it is just it's you know it's looking for those one two percents because that's
typically after you found the broken things that's that's all you can hope to achieve but if you have sufficient scale
then yeah it's um it's absolutely worthwhile that's really interesting because i and
it's worth saying that at the start actually when it comes to the sort of split testing and a b testing you are talking small percentages because i
think there is an ex an expectation when you go to certain software's home pages that you're gonna
you know percent uh increase in sales increase in sales
and these are definitely the exception and not the rule and the common theme then with cro certainly as you get as
you get bigger and you your longevity is there you've been around for a little while the numbers become smaller is that
is that right yeah absolutely and i think those i mean those numbers uh uh i think they sort of plague cro as an
industry really um and we get them from all angles you know the the software vendors absolutely are incentivized to
talk about those very very rare experiments i mean they do happen they do happen where you change two words and
you know conversion goes up by five percent or whatever absolutely but they happen very very rarely but of course those are
the things that uh the vendors talk about those are the things that agencies talk about
and it sets this sort of burden of expectation where everybody thinks that's what they should be getting and
often that's how programs are sold in or if a business decides to work with an agency um they've all heard these
incredible stories of people you know changing the button color and getting the double-digit conversion rate increase they might sell the agency
relationship into the organization on that basis or they might build the business case for establishing a program
on that basis i work with a very large retailer here in australia who has set
up a you know a big experimentation program but the whole thing was founded upon this idea that percent of all of
their experiments would create a winning result which gave them an uplift
and they've got scores of people working on this and the problem is of course they're not hitting that there's there's
benchmarks there's a great slide i saw somewhere once which showed sort of benchmarks of publicly available data
from companies that have big established optimization programs so airbnb
google booking.com um you know all the biggies and generally i think their strike creates
probably you know it was i mean differently reported numbers but let's say
at most five ten percent and that doesn't include just getting a winning result that's getting a negative result
as well because most of your tests will do absolutely nothing that's fascinating there is and so
there's there's there needs to be i guess dare i say it we don't like to use this
phrase when we're entrepreneurs but that almost needs to be a common sense approach to to cro then which says
actually let's let's have a little bit of realism and we don't like realism uh do we we just we just don't we we we
like the big hairy audacious goals and and and that sort of thing but actually
there's there's a winning formula to be found in the sort of the actual realistic numbers
and once once you're i think uh to quote jim collins brutal brutally honest with the with the facts
um actually you can then start to do some really interesting stuff so what are
some of the things that you've seen work well uh in this space the biggest the most effective way to
optimize your website that i've just seen again and again and again which hardly anyone does because it's not cool
it's not sexy it's not data driven really or people don't think it is is
literally just user research it's sitting down talking to users showing them your website setting some
tasks and writing down what they think doing that a few times and sort of matching the themes
it's just brutally effective it's brutally effective but it's not sexy
not very sexy no no i mean no one likes doing it now people will come up with all sorts of excuses to just not have to
talk to their customers in my experience yeah that's really interesting okay
because this is again um i mean this is why we've titled the podcast you probably don't need a b testing this is
what came out of our conversation in the prequel we were talking a little bit about a b testing but you're like well
actually you probably don't need it especially if you're just starting out in e-commerce um you know or or or you're quite new to
the game or if you haven't actually done customer research before and right and so this sort of seems to be
um from what i what i uh have taken away from you all of it this seems to be by far the first place to start
when it comes to your cro when it comes to optimization so why is it
then if this is so brutally effective and again a great phrase um
why is it that we don't do it why do we go to great lengths to try and avoid talking to our customers
yeah it's interesting i think there's there's an awful there's an awful lot going on there um partly i think it's just it's a
little bit messy you know you have to go and sit down with a real person that you
have to find firstly so you have to have to engage them you have to sit down with them you have to get them to
show you how bad your website is oftentimes you know and people don't like that it's
very uncomfortable there's a there's a great quote from avinash kalshik uh google's uh analytics evangelist i think
his title is that used to be on usertesting.com which is a great um now quite enterprise focused but it's a
unmoderated usability testing platform and there was a quote from him on their homepage and it said
run the test look at the results cry make a million dollars
like that okay but it's i think it's hard for organizations as well to act on what is
essentially bad news um one of my former colleagues a ux researcher and designer called
harry brignal who's um an interesting guy who instantly came up with the idea of dark patterns which is another story
but if you're not familiar with them i encourage you to explore them he wrote a great blog post dark patterns yeah these
are the things that um you know sort of ryanair uses to trick you into spending money that you didn't intend to spend on
your um on your cheap fare okay sort of ux ux hacks
um he's written a lot about that but this is unrelated he talked about um in a great blog post he talked about
i think it's called what makes user research unique and he said the thing that makes user research unique is it
brings bad news it's painful it's telling you all of the things that you've done wrong all of the
assumptions you've made about your customers everything that you've got wrong and people just people don't like hearing it
i think that's fascinating we do we do it is
within our human nature to avoid bad news we don't like confrontation uh you know we we
we tend to avoid things that sort of give us the bad news don't we any
and so i guess i can see why this sort of seeps into uh you not doing
user research because we it is messy it is complicated you are fundamentally dealing with people the thing you people
like about software i suppose is it is it's software right and it's just over there it's
doing it's not personal it's just doing its thing and you can measure how many people go to page and you can measure how many
people go to page b and that's a beautiful thing and i don't have to get involved uh whereas user research like you say
the first question is always going to be why why why do i don't talk to people that's why i got
into e-commerce i could avoid people i mean there is there is this sort of
almost mythical ib that i think launched a billion terrible ib tests which is um
was written up in in wyatt a few years ago so it's where google famously couldn't decide between
color a which was a colored uh blue and another shade of blue for their link colors so they famously ran this a b
test where they they tried every shade between them and the results apparently led to a sort of you know multi-billion
dollar increase probably by now in click-throughs and i think that's the myth on which a b
testing is often sold people think it's you know you sort of throw your site into a blender it breaks it
algorithmically down into all of its component pieces and it throws them all sort of back as different variants and
magically it spits out a double-digit conversion rate increase which is absolutely not what happens you know the
best even if user research is the best way to find out what needs to be fixed about your website a b testing is really
the best way to validate that at scale this is this is the sort of drama i'm always banking on even with clients that
have sophisticated a b testing programs often the reason why they're not getting the sorts of results that they expected
is because they're not doing the research to actually find those concepts in the first place which really just has to start with qualitative research
okay so i get that it's messy i guess that we don't like um bad news
are there any other reasons why we tend to avoid consumer research
none that i've thought of particularly i think there's sort of this you know different overlapping versions of that i mean in
large organizations people will say that it's it's hard for them to talk to customers there's often sort of layers
of of bureaucracy which are involved um people have a perception that it's expensive which it can be sometimes if
you use um recruitment agencies um but i think it's typically just that
it's um you know it's uh it's a bit messy and demanding and perhaps something that is
uh outside of the skill of most most people you know it's a particular skill to be able to sit down with somebody and
try and sort of tease out the themes and common threads uh around you know why a
website is you know difficult off-putting hard to use whatever it is
um but yeah i think that that really captures it for the moment very good so uh don't go anywhere we're just gonna
take a few moments to listen to or to hear from the amazing sponsors that we have which enables us to keep doing the
show and enables us to talk to amazing people like oliver we'll be right back uh following uh this from our sponsors
did you know that nutrition is one of the keys to maintaining the energy you need to drive your business forward
vegetology creates incredible unique supplements in an eco-friendly ethical
and sustainable way that feed your body with the precise nutrients it needs
we're not just making you healthier we're helping to protect our planet too our products are vegan friendly and
approved by the vegan and vegetarian society plus they're gluten free so they fit perfectly into any lifestyle they
also contain no artificial colors or flavors making them good for your taste buds too you can feel good about your
food choices with our healthy natural supplements we have something for everyone whether you want to boost your
immune system or just get more energy every day and we're always working on new ingredients so that we can provide
even better products in the future so what are you waiting for get started now by heading over to vegetology.com
[Music]
hey there are you a business owner here at oregon digital we know firsthand that running an ecommerce business can be
really hard work as the online space gets more competitive it is becoming even more challenging to stay ahead of
the curve we totally get it so we want to help you succeed by offering a wide range of services from fulfillment
marketing customer service and even coaching and consulting just so that you can do what matters most save yourself
the time and the money and let us handle the day-to-day tasks this way you can run your business without having to
worry about the boring stuff so what do you say are we a good fit for each other come check us out at oriondigital.com
and let us know what you think
so i am back with oliver we are talking about conversion rate optimization uh
and why you probably don't need a b testing so let's dig into that a little bit so we started talking about a b
testing we defined what it was we then said you know that user research was brutally effective
um so why do i probably not need a b testing why
why why do i need um consumer research instead of a b testing if that makes sense
yeah so i wrote a blog post um probably last year that was called you probably don't need a b testing and the reason
why i wrote that was because i often um got you know prospects coming to me by
my website wanting to start a b testing and i really got tired of telling them
the same things um so i mean the reason why they don't need it is because they probably don't have the
traffic and that there is more effective ways that they can go about gleaning that insight i think it's almost like a
way you sort of graduate from user research qualitative observations about your
users and then once you've got sufficient scale you'll graduate into validating that with with quantitative
research via a b testing but you certainly can't have a good a b testing program without a good qualitative
research program of insights feeding into that so you may as well set up that foundation
to begin with so this so this is in effect for any company is
this your first port of call is is consumer research before even before a b testing yeah absolutely i mean
there'll be a certain amount of um of quantitative research and analysis so digging through analytics data as well
that's obviously a great way to find opportunities looking at the conversion funnels looking at the traffic mix and
so on seeing where good traffic's coming from where bad traffic is coming from but it certainly needs that qualitative
lens on it as well so the
the ideas then behind so when you say consumer research or customer research i i i just say user research by the way
okay so user reset okay but let's get our phrases right uh so when we talk about user research in my head right the
default picture in my head and this is very wrong and i'm maybe it's very right olivia you're going to correct me
is i'm stood in a room in a lab coat with a clipboard and i've got four or five people busy
using computers and i'm just sort of going around taking notes uh as as when they raise their
hands um i assume can user research is very very
different to that so how how does it actually work in practice so it really depends on i suppose on the
size of the operation sometimes it can be you know it can be very informal
i was working for a telco a few years ago in the uk and my user research was
uh going into their stores and sort of listening in on the conversations that the
the retail staff working in the stores had with customers that was a way for me to understand what were the sorts of uh
you know the concerns objections and things that customers had so that's a that's a um
you know a fairly particular form of of user research that was easy to carry out in that case because they had a store
actually very conveniently close to our office it didn't require input from anyone to do that
sometimes it's been even one of the retailers i work with here in australia some years ago i used to set
up in their outdoor furniture department with a stack of gift cards and as
people would walk past and say hey can we run talking to me about um this website for a little bit um people were
universally happy to do so and simply just try and probe and ask some questions and find out what were the
things that they found confusing difficult and off-putting sometimes it can be um it doesn't have
to be in person of course over the last few years it's very rarely been in person but in some ways one of the good
things about this pandemic is that people have become much more adept at using video conferencing software so
typically these days it will happen over zoom and we'll sit down with sit down with users not so many it will
be individually one at a time typically about five users per round
and set them some tasks normally say you know visit to visit this website
they will typically have been targeted through that website so something that's actually quite new to
me which is has been a real game changer is one of the ux researchers that i work
with i asked him to compile a list of prospects from
existing customers from email addresses and it seemed fairly straightforward to me
and what i'd always done in the past his approach was well we know that
of people coming to the website aren't buying those are the people we want to talk to which is one of those
things that's maddeningly obvious in hindsight and so he created an intercept that sat
on the site um and people would say hey we want to talk to you you know we'll give you a voucher or something and so
people opted in and uh we would get people that had never bought from the site before
and then um yeah set them some tasks say you know on this website um find something you like go through the
process of buying it that's a way of just doing a real sweep for ux issues
you know we might find that the filters aren't easy to use or um you know something like that something very basic
and then we'll sort of go in and probe and find out a little bit more around you know why they purchase
um that particular product category online for instance and
try and understand more about their consumption habits and typically i mean fascinating things crop
out of this i did some work recently for a for a wine retailer and
they sell mixed packs of wine online and what we discovered through this research
was that people have a um mistrust of best mixed packs they see
that as their sort of offloading bad stock but they did not have okay any
mistrust for a subscription everyone said wine subscription i'd sign up for
that so we can change the wording and make it a recurring billing and suddenly it's um it's a much more profitable
endeavor so that's just one of the just really arcane and interesting things that you know you find out when you sit
down and chat to your customers that's fascinating that's fascinating okay so there's a lot there
so if i'm going to do some uh user research um i like the idea that you had
of talking to people who are not buying as well as the people that are and you're right i mean there's a you know
when you look at the stats on your website whatever the numbers are you know at percent of people are
not buying so these are people that actually are very good to do some uh research with so
how do you what sort of different if i'm running an e-commerce business what are some of the
methodologies maybe that i could use to try and bring them in i mean you talked about standing there with a gift guard
maybe we're doing a pop-up type of thing what sort of thing works well to get people to give me the time to do the the
research yeah i mean i think it's just an inducement i think it needs to be quite a healthy inducement as well
i think in this case this was a this was a london-based retailer we said i think it was pounds or something for for
minutes or an hour of people's time um and that was sufficient and in my experience it's really it's it's it's
better to um you know spend more money basically offer a higher inducement and that
really commits people to a showing up you get a much better um you know not so much of a notion a lower no-show rate if
you offer a better inducement but also um i think people feel um
somewhat like they um you know they owe you a sort of a depth and a quality of response
that's that yeah it's and actually your value you're showing that you value them in their time and i think there's always
that reciprocation isn't there with that um so okay so you could do some kind of pop-up i suppose which actually says
listen uh quid for you know minutes of your time sort of thing now
without getting too nitpicking into the detail of um minutes is that about what i need
yeah i think so i'd say minutes to an hour yeah i'd typically allow an hour uh
per per respondent yeah and so each responder that comes along they're
coming along on zoom and you can use services like zed cal or whatever to sched so they can schedule
appointments that work for them they're on zoom i'm assuming um they're sharing their screen yeah and you're
watching what you're doing and you're recording uh the whole thing so you you've got to
have their permission i guess to record uh the call um and so they're sharing the screen so
uh i i i'm there in my head i can picture what's going on um so at this
point you mentioned that i need to give them a series of tasks uh to do
um is is it as obvious as saying i want you to
buy our best-selling product or is it are we giving them quite complex tasks so are we giving them multiple things to
do on on this on the website during that hour yeah i mean i think you wouldn't spend an hour
simply in task mode i think but it really depends what you're testing if you're running a very general test a
sort of a first sweep of your website if you're not testing you know a particular new feature or something like that and
you really want to understand what the the blockers are then i would think a general task that says you know find
something that you like from our website for instance and go through the purchase of process of purchasing that because
that will allow you to see how do they interact with it do they use search for instance do they use your category pages
do they use filters how do they interact with the the drop downs on your product pages so within that task you really
have the i suppose the discovery aspect of it and then also the transactional bid as they go through the checkout
funnel okay so um
so i'm watching them do this and i i'm very aware that um
that i guess the words that i use uh could lead them one way or another and so i have to try and be as independent
or as neutral i have to be like switzerland i suppose when i'm conducting the research you really do which is it's it's very difficult and
that's really why somebody who is a little bit sort of impartial who hasn't made those decisions who's not connected to the
website is really the best person to be running that user research i i signed up
for my power company asked me to participate in some research about their app recently and it's a
small sort of startup power retailer based here in melbourne and the guy who ran the
research sessions was a sort of hybrid ux designer ux researcher
and i was critiquing his decisions and he got very defensive and it was
almost useless we shouldn't have been he shouldn't have been running that session at all
so someone independent if if at all possible from the site should be running
running that so and i guess so
and this again i guess it's all going to come down to budget isn't it because if i'm if if i'm uh a startup if i'm
hustling then actually i'm probably going to be doing these user research reviews myself because a
lack of budget is going to stop me getting someone independent to it especially you know like yourself a consultant or someone with experience in
this whole area and so can i still get meaningful results if
i do it myself uh they're maybe not as good but is it still worth doing if i can't get an
independent person to do the reviews i think so i think it depends on how on how impartial you can be but yes i think
so the other thing that's worth mentioning that we haven't really discussed yet is unmoderated user usability testing so that's using a site
um the original one was usertesting.com i think there's user biller there's user zoom maybe there's lots of these
different sites out there if you google unmoderated or remote usability testing there's a slew of these sites and in
that case you actually don't have to be there you just watch you watch videos so you set
the task and you say you know find something on our site go through the process of buying it and
what will come back is a screencast and they will you know these participants are very good at verbalizing their
thoughts aloud as they go through the process the first time i used that was really it
was using usertesting.com and it was many years ago now it's the very first
um yeah user user test i ever ran i was working for a magazine um a site that
sold magazine subscriptions here in australia yeah and we just set up a normal sort of
sweep you know we said find your favorite magazine and go through the process of buying it um and
people got to the everything was fine until we did five of them five out of
five customers got to the checkout page they saw a notice that said you will receive your first
magazine in eight to twelve weeks which is exactly how the magazine industry works because of lead times in
publishing the fact that you've got distributors using you know old dot matrix printers in warehouses somewhere
there's imports there's all sorts of delays so magazine subscriptions certainly back then took that long they
probably still do so we thought we were being like amazon just telling people exactly when they
could expect to receive their first issue all of the people that we tested said this site is a joke absolutely no way
would i buy something that's going to take to weeks to get to me we checked all of our competitors and
saw that we were the only people that said that even though they were using the exact same distributors same magazines and had the same issue
we very cautiously took it off expecting a flood of inbound complaints which we
we watched very carefully and conversion went up i mean a lot like percent or something almost immediately
which is just one of those things that we never would have seen you know we had the curse of knowledge um you know
they're saying what you can't you can't read the jar if you're if you're inside of it essentially
yeah that's very true yeah i like that i like that so there are these sites then that we
can use um unmoderated user testing sites and just google that and there'll be a whole and i'm guessing that the
people that do the user testing are experienced in this area hence the reason they're
used to verbalizing what they're doing and what the problems are and so on and so forth it's one of the reasons why i think it's it's not quite as good
actually because you would typically get sort of quite professional testers
and you know they won't necessarily have an affinity with your product category with
your site and i think often things can go undetected with remote usability testing
versus in person which gives you the opportunity to really probe deeper but certainly it's better than nothing and
um you know hopefully if your listeners tried it might might get them hooked and help them sort of see the value yeah and
it's certainly a good place to start i suppose um but again like you say there's nothing that beats
actually talking to the users on the site um and figuring out what's going on with them so i'm assuming in this
conversation that's like minutes to an hour there's a mixture of tasks that are happening i want you to do x
for me and just talk me through whatever's going on in your head as you're doing it
but also i imagine you have a series of questions just like general questions which aren't
necessarily related to the site but maybe are related to the user and whereabouts are they and uh you know
sort of simple user information um is is that is that right so it's a
mixture of sort of um tasks and just general conversation and
just seeing what comes out in that conversation yeah i mean sometimes it's general conversation sometimes you're really probing to try and understand um
the i suppose the deeper trends or the the the things that are the things that make people want to go
online and buy that product or not for instance um you know i think if the you familiar
with the idea of jobs to be done uh go ahead and explain it right so jobs
to be done is something i know um clayton christensen the guy who came up with the the innovators dilemma was um
was involved in this and some research that looked at the reasons why people buy things and
the example they talk about in this book is doing some research on customers buying
milkshakes in the u.s and what they what they came to realize through this research is that commuters
buying milkshakes are not buying milkshakes because they're thirsty or hungrier because they want something to
drink it's that a milkshake in a long commute is something that keeps them from being bored so the job to be done
with a milkshake is not sustenance it's not um quenching a thirst it's giving me
something to do while i drive and stopping me from being bored and that's often the sort of thing that we're trying to find out with when we probe
deeply in this sort of research to find you know what is the job to be done with this thing why are you visiting the site
why are you buying this thing online versus uh in person um you know what are the really the
deeper things that sort of sit beneath the obvious and so that just comes from asking
questions and probing uh the the users that come to the site so i
i've got the user testing i've i've had some great conversations i've got some insights i've watched them uh you know
using the website oh my lord i need to change this on the website okay cool how do i record that data is there is it
just a case of i just wipe it into a google document or is there is there a better way to sort of help
myself in terms of recording the data that i find yeah i mean it's a bit of a it's a bit of a holy grail i think for
many organizations certainly for large organizations everyone that i've ever worked with has had their own system or
their own plan for how they're going to create a single repository for all of their research that will be indexed and
searchable and and whatever my feeling is that for the most part it should be disposable you know you will
if you do say five interviews you will notice some things very quickly i think um uh you
know that you'll need to action i think it depends on what you want to do with it if you need to to write something up
to convince people within your business that they need to act on it then yeah it's a matter of synthesizing into you
know what those themes are and how prevalent they were within the users that you spoke to but for the most part
and particularly if it's if it's you know very early on if you haven't done that sort of research before it's going
to be blindingly obvious what you need to do you're going to say oh right people don't understand how to use the
filter you know we need to fix the filter for instance yeah that's that's great and i i'm
i'm with you i'm just like whatever works right it's it's honest now you've mentioned this a couple of times and you're not
the first guest to mention this but you keep coming up with this number five right uh
which i think is quite interesting explain some explain to the listener uh
why five why five users five is i mean this is contentious actually so this is
something that comes from the nielsen norman group the the sort of godfather of web usability is a guy called um
uh god not don norman but his partner jacob nielsen um he was one of the first people you
know back in the mid s that was talking about the usability of websites and has done a lot of work um you know
promoting ux research and usability and so on and he wrote a sort of a famous article some
years ago which uh talked about the fact that in their experience percent of all usability
issues could be identified with by talking to five users
subsequently it's been very contentious a lot of people dispute this i've just latched on to the idea of five you know
five is better than nothing i think it's better than four it's not quite as many as six it's a kind of a neat little
number of users to talk to um but the point there is it's not statistically significant you
could talk to a hundred users one of the things he says in the um in that post he says that people often hire us big
companies hire us and say we want you to talk to users because that's going to be statistically significant people
are going to pay attention to that number internally if we say well we spoke to five users and we saw this it's
not going to carry any heft but the point here is this is not science it's
business so let's just talk to a handful of users and see what comes out of it and five is as good a number as any i
think yeah no it's and actually five is quite i'm sitting here thinking
um of my own e-commerce business like and five actually is quite manageable mm-hmm
jeremy you can you can look at five and go i could do five yep um uh that's
that's not too much of a problem i guess one of the questions then um about this whole idea around uh
customer research is that you continually and perpetually do
or is this is this a one-off event because five would would
would dictate to me a project it would say that there's a start date and indeed whatever the number is whether it's four
five six seven eight nine hundred or whatever it is that's a project with a start date and an end date so should i be repeatably doing those
projects throughout the year um i think it depends on you know what
your um what sort of available what resource availability you have um and to what if
you keep finding new and useful things then of course you should continue to explore um many
people may find that they hit diminishing returns you know fairly early on but i think it will it will
really vary and it can be you know um one of the
a really great project i worked on once was with a newspaper over in the uk we were sort of redeveloping
uh the the whole website we worked in a in a team of a small team whether you
know a designer a developer a ux researcher i was doing optimization product manager um and we would
run on these one week long sprints which culminated in what we called testing tuesday so every tuesday everybody would
sit in a meeting room and we would watch a usability researcher sort of two meeting rooms up um talking to somebody
and showing them all the work that had happened in the week before and that was something throughout that process of
redevelopment that happened every week for months and months and it was a matter of you know initially
all of those sort of facepalm moments and then you know they'd fix the thing and then gradually it got better and
better and better and better before it was released so on a big program you know project if you're working on a
re-platforming or something large like that absolutely regular iterative usability
testing can be great for most people i'd say you probably will hit diminishing returns you know
after you've done it a few times but just check in periodically because you make changes you know websites are constantly evolving so
um you know go in validate those things you don't even need to show people your
actual website if you're thinking about making a change you could show them a clickable prototype that you can make in
keynote or powerpoint or something you could show them a sketch you know it depends what you're doing but
whenever you are making those decisions and changing things i think there's um there's some value to be had running them yeah no fantastic advice and um
yeah thank you for also explaining the number five both you and aj davis talked about that and i'm like okay that's
fascinating that this now i understand why uh which is very very helpful
um oliver i guess
in an ideal world right um as a e-commerce business
owner myself we have an e-commerce website um
i get that i should be doing some form of optimization it makes total sense to me right then
and i get the arguments for it um i am now convinced uh between you and aj that
actually um i need to be doing user testing uh on that
um and i need to think about that long before i think about a b testing i guess in an ideal world is it fair to
say that i should be running both uh both those things is there anything else
from an optimization point of view i mean i'd not necessarily get into detail just from a headline point of view that i
should also be thinking about um i mean i think your analytics is um
is is you know a big part of that i think people in in the world of e-commerce are fortunate in that
more or less out of the box if you're using google analytics you can have very good tracking to actually
understand um you know the changes that you're making the the merchandising initiatives or whatever else you're
doing on your website you can see very clearly what they're doing in terms of impacting on on propensity to purchase
but yeah i think that's a critical thing to to have set up and i'm surprised
actually how often i look at look at analytics implementations for
small and sometimes medium-sized businesses that actually haven't enabled some of the critical features in there
the things that you really just have to tick a few boxes like setting it up so that your google analytics records what
people type into the search bar for instance uh the you know the on-site search function so many people don't go
through those sort of basic hygiene setups of ensuring that they've got that set up or e-commerce enhanced e-commerce
functionality implemented as well so if you don't have those i really recommend and it's all there isn't it it's not it
like you say it's just a case of switching it on and making sure that you're tracking that data
i mean certainly in the case of using a you know a shopify or woocommerce or something that they are pretty much out
of the box um set up to implement a data layer and yeah you just turn on those features fantastic fantastic so user
research a b testing and analytics good core functions of optimization the
holy trinity the holy trinity of optimization i'm
sure somebody somewhere has drawn like a venn diagram and there's three insect in circles and in the middle where they all
intersect you know it's like the holy grail is is right there in that in that little bit right
there's a dollar sign in the middle yeah fantastic oliver listen i really enjoyed our conversation this morning for me uh
this evening for you um how do people reach you how do people connect with you if they want to find out more
yeah i'm at oliverpalmer.com uh blog there periodically and there's links to linkedin twitter all that sort of
business uh there oliver that's pretty easy oliverpalmer.com do check out oliver's website i'm sure he would love
to hear from you and of course you can get all of uh well that one single link uh we'll put the link to oliver's
website uh and uh his linkedin profile if we can find it um all in this week's
show notes uh where you can get the notes transcripts and all that sort of good stuff so if you get those delivered
to your inbox automatically that will be in there as well so oliver thank you so
much for being with us really really appreciate your time and your insight uh and yeah it's interesting actually
how uh user research is now becoming a bit of a hot topic and more more people are wanting to talk about this so thank
you for your insight thanks for giving us the head start thank you very much matt it's been great
to be here well a special thanks to my guest oliver what did you think
did you learn stuff of course you did it's just really helpful to get that information isn't it and if you're like
me you kind of you you rapidly take notes uh from the guests and you always
think all they could be better so if yours are like mine and they're kind of scrappy notes well just head over to the podcast uh page on the web for free you
don't need an email address or any other sort of stuff ecommercepodcast.net forward slash you'll find everything there the links
the notes transcript everything we've done it for you yes we have uh such a legend isn't he now
as i said this is episode at the start yes i think it deserves a celebration and maybe if the sign guys
are paying attention they can put in a little celebration music right now cha-ching yes they can [Laughter]
i really hope they do that now next week then if this is episode
next time well that's episode we've got something special lined up for you yes
we have so i'm not going to play you an excerpt from next week's show uh i'm i'm not i'm not going to do that no i'm not
going to play you an excerpt from episode i'm just going to let it all be a huge surprise for you yes it's
going to be epic yes it's going to be large yes it's going to be worth the episode so i just can't believe we
got to episode already amazing really i am really looking forward to this it'll be a little bit different but
do stay tuned of course and if you haven't done so already subscribe to the podcast follow it
like do all the things that you need to do to get it to auto magically appear in your
inbox uh or on your platform of choice whether that's youtube apple google amazon stitcher
spotify we're on everything i think at the moment so you know you can use it and whilst you're there subscribing just
leave us a little rating or review just helps us to reach more people which is a beautiful thing because
without you our fantastic audience we would never have got to episode that's for dangshore so uh we really
appreciate you being with us on the journey and the fact that you guys are so helpful in spreading the word about
what we're doing is just a beautiful thing so do keep doing that i really really appreciate it let us know how you
get on with the podcast love to hear your feedback uh as i said at the start from today's show episode
go to ecommercepodcast.net forward slash and get all the notes and stuff for free
thank you so much for listening honestly it's great to be with you it's great to learn how to deliver e-commerce well i
love doing what we're doing i love talking to the guests i love the show it's brilliant absolutely love it and if
you don't get anything out of it i know i do which is one of the reasons why we keep doing it yes it is absolutely so
that's it from me i think i've waffled on enough i will see you next time in episode
you've been listening to the ecommerce podcast with matt edmondson join us next time for more interviews
tips and tools for building your business online