AB Testing for Optimized Performances

JÉRÉMY: 0:01

If you think about it, actually sales and marketing are pretty intimately connected. It's all about the same. The ultimate goal of marketing is to generate leads for the sales team. So to increase the sales, you know, when you have a subject lines, you see that your open rate is not good enough. And so maybe you want to work on the preview lines and the subject lines to increase that open rate. Essentially that a b testing. It's a method for making informed decisions across the different possibilities. And that's the beauty of AB testing. Provide concrete insights to optimize conversion, user experience, and making strategy based on real user behavior.

ARTHUR: 0:36

Hello my dear Odooers was ready for another episode of planet Odoo. I'm your host, Artur Karia, and today we've got Jeremy or international sales representative at Odoo. Together, we'll dive into the interesting discussion of AB testing for optimized performances. Not only will we explore the significant improvements this method can bring, but will also delve into the why behind choosing it. This episode is your abecedarium to the AB testing, so let's go! Hey Jeremy, for today's podcast, could you please introduce yourself to the audience?

JÉRÉMY: 1:09

Yes So I currently work at Odoo in the sales department and I have a background in marketing. I started my career in a marketing solution in a sales position, and along the way I got the opportunity to collaborate with major marketing agencies, media brands. So different entities.

ARTHUR: 1:25

And with all your specific skills etc. that you gathered through all this experience. How do you juggle? Because sales and marketing could work together. But how do you do it?

JÉRÉMY: 1:35

Yes, during my experience, if you if you think about it, actually sales and marketing are pretty intimately connected. Um, it's all about the same. The ultimate goal of marketing is to generate leads for the sales team. So to increase the sales. So when you think a bit deeper, um, all the communications, the, the different channels that you, you provide, the, the segmentation that you do, the unique goal behind that is to, to find a good communication to increase the sales. So sales and marketing that's intimately the, the jungle. And I could say, uh, kind of personalized marketing if you if we speak about marketing.

ARTHUR: 2:14

I would say they are brother from another mother.

JÉRÉMY: 2:16

Yes.

ARTHUR: 2:17

They mix together very well if you use it correctly.

JÉRÉMY: 2:19

Exactly.

ARTHUR: 2:20

Now let's start a bit with more overall questions. Let's dive into the subject of AB testing. What is AB testing and uh, what's the purpose of it?

JÉRÉMY: 2:29

What is AB testing? That's the big question. And yeah, it's uh, it's a powerful strategy actually A B testing in marketing and product development in various areas actually. So if you have, for instance, an email, a web page, an app, and you want to choose which version of the email is actually the best one, so then you will drive the the different actions afterwards. So you create the variation. So the idea is that you have a version A, a version B with some differences. Uh, like if we take a concrete example, you want to change the, the color button. And so with the the interactions you, you want to make sure that you choose the best version for the final one. Uh, but in email marketing in a web page or in a, in a product. So then it's all about making measurement using metrics and determine which version performs better. And that's the beauty of AB testing. Uh, provide concrete insights to optimize conversion, user experience and making strategy based on real user behavior.

ARTHUR: 3:30

So so basically it would be like making the games, uh, for your kids about spot the difference. Right.

JÉRÉMY: 3:35

Spot the difference for kids. Yeah. Yeah that that could be. Yeah.

ARTHUR: 3:39

But like yeah. Uh, with metrics and like what works and what doesn't work. Yeah. That's interesting. Absolutely. And and so you can use AB testing for multiple reasons and not only marketing then.

JÉRÉMY: 3:49

Yes, absolutely. It goes beyond uh marketing. It can be applied to various different aspects. So website product development. So you have such more uh possibilities to play with AB testing in different areas. And the application are way broader than marketing. Um, they can be applied to enhance user experience on website on the apps testing different design elements. Uh, also in product development. So to choose really what are the best features to develop next in the product evolution. And even if in email campaigns, uh, if you want to refine the the subject lines, you know, when you have a subject lines, you see that your open rate is, uh, is not good enough. And so maybe you want to work on the preview lines and the subject lines to, to increase that. That's open rate. So that's essentially that AB testing uh, it's a method making informed decision across the different possibilities.

ARTHUR: 4:42

I just have a little question about maybe opening rate. I know you cannot apply it for everything, but could you maybe give me an input about what is a good opening rate? Depending of what you are doing with your AB testing or just an email?

JÉRÉMY: 4:56

It's a really difficult question because the open rate. For a specific company in a specific industry will not be the same as another industry. Completely different. So a good open rate is difficult to say if you have in general an open rate between 25 to 30%, it's already a really good open rate, I would say.

ARTHUR: 5:14

Okay. And so also if you are a niche company, the advice would be don't try to compare to other statistics at some point, because you are totally different from a broader and wide company that could provide, uh, services that everybody uses.

JÉRÉMY: 5:30

Yes, that's difficult to compare, and that's a bit the tricky in the in this game you need to really know your own customers. Your own customers are not the customers of your competitors or even the other solutions that are completely different. So get to know your own customers. And it's important, more likely in marketing to segment your audience to address the right message to everyone. We were discussing about sales and marketing, so when you are in the sales process, you want to make sure you cover the needs of your specific person in front of you. And the client is not another. So you cannot do that with when you have a multiple contact on a on a mailing list. But what you can do is to identify the segmentation, uh, and the different segment of your audience to have the right argument to each segment group. And yeah, if you provide the good information, then, uh, normally you could expect better open rate, better click through rate because that's just relevant for your audience.

ARTHUR: 6:26

Thank you very much for that. So to to get back about the subject and thank you very much for your input. What are the benefits of email marketing then? And how can we use AB testing to improve it?

JÉRÉMY: 6:36

Yes, in email marketing actually it's a it's a really powerful tool. Email marketing still in 2024. Now to uh, to engage with your audience because first, it's a direct communication to your audience. You have the ability to choose the message you want to send to the specific contacts. You have the power, actually, so you can generate a high return on investment thanks to that. Uh, it's not, uh, really costly. And you have the, the control on the on all the content, all the relationship we want to build with, uh, with email marketing, thanks to the email marketing.

ARTHUR: 7:07

And so how to get things to click and how to refine your AB testing so you get the best rates and just. Yeah it works.

JÉRÉMY: 7:16

Yeah. Just actually the the AB testing in email marketing is interesting because what you want uh on when you send an email is that your email. Uh get all the attention increase the open rates have an impact actually. And that's that's really a B testing comes into play actually, uh, with two variations, the version A and the version B, you will be able to see on the first sample of your audience. What is the tendency, what is the email that performs better regarding the goal that we set. And so after that you just decide which mail you will, uh, you will send actually to your audience. And uh, and that's there the magic of the, of the a b testing in email marketing.

ARTHUR: 7:56

Okay. And uh, when you start an AB testing campaign kind of thing, what are the main steps when starting it? You told me when we prepared the this, uh, podcast, there was, uh, four steps or even more like ideation, hypothesis metrics and fallbacks and of course reporting. Could you extend a bit this.

JÉRÉMY: 8:16

Yes. You you need first to know what you want to test. It doesn't really make sense to test different elements at time, saying, okay, I will change the pictures. I will change the subject line. Then you are not able to to concretely know what you are going to test here and what is the impact of your test. So step one set a clear goal and uh, define what you want to test. Then you will formulate hypothesis. So the idea is to predict a bit to have an idea of okay, normally if I do this this will improve the the open rates maybe. So have some hypothesis meaning that your anticipate a bit what will be the results. Then you concretely manage the different version the A and the B, and you identify the metrics such as the conversion rates, you you, but you keep the focus on the, the, the main goals. And then you will check the secondary metrics like bounce rate. And it will provide a broader perspective in this case. So during the testing period then you need to analyze the the results and analyze the statistical significance of the of the insight. And after that you use the results to iterate and make informed decision based on the results concretely, and not just hypothesis or guessing it.

ARTHUR: 9:30

And uh, would you have an example of an AB testing like, uh oh when you start it? And yeah, an example.

JÉRÉMY: 9:39

Yes. For example, um, you are drafting an email and uh, you it's a promotion. You will make 10% discount on all sales. And so you will communicate that, uh, obviously it's an important moment for the promotion for you, and you want to make sure that the email is the the best one for, uh, for to promote the promotion and to. Uh, travel the traffic to the website. So you will create the first version of the email, and this version will be straight to the point with an image. Uh, maybe you can personalize a little bit the content, actually. And yeah, you set up your first email with the clear message 10% discount on all sales. Set up the date. Uh, and, uh, no, you want to create the B version with something more graphically interesting with, uh, yeah, maybe the different images that are more engaging, something not straight to the point to the fact that there is only the 10% discount, but really on the fact that, yes, you have more, um, visibility on the mail, more, more, uh, a better layout. And so you will create the version B with this, uh, specific design that is different. And you will see on the small, uh, sample size. So small sample size mean, but I mean also a significant sample size. It needs to reflect the reality of your audience. And you will send the A version and the B version to a specific percentage, let's say two, 3% of, uh, of your audience. And you define afterwards, after a testing period, test enough to be able to, uh, to analyze the results. And you will define which versions perform better in terms of click rate, purchase and so on. And that's the magic of AB testing. So you will make some small changes one by one. And you will be able to know which one performs better and learn the results for the next one.

ARTHUR: 11:29

And with all the data you gather through those experiments, let's call it experiments. But you get the point. Um, can you do AB testing and at any time or like, yeah, how do you organize it?

JÉRÉMY: 11:41

Yeah, I often learn uh, and Earth. So yeah, that's the theory. But in the reality where we when we are in a marketing department, we are already sometimes rushing to finish the version a, a single version actually on time. So yeah, it will take time to, to make a second version of the mail. And then I need to analyze. And so the idea behind that is that you really need to understand that the time that you will spend on the campaigns, you will pick up the fruit afterwards. So it's not a loss of time. It's really a time that you will spend to increase the performance of your emails or the different tests that you you can make on the website. And so you will just pick up the fruits, uh, afterwards. So, um, yeah, that's, that's a bit, uh, where a b testing is interesting. You need to, to invest the time, not lose the time to, to make it happen.

ARTHUR: 12:30

And about AB testing. Can you in some way summarize why it's really important and could really help boost some business? And also how can it help to make the right decision?

JÉRÉMY: 12:40

If I say it in one sentence? Uh, it's crucial because brings brings, uh, uh, a data driven approach to decision making, because by systematically tests the variation, you gain insights on what really works. You start step by step to guess what, uh, what works for your audience, validate it with results, adapting it in your next communication and one campaign after the other, you start to really know your customer segments. So that's an empirical understanding that will help you to make the right decision. When you will optimize the different pages on your website, on your emails, on your app. So AB testing is a tool that you need to use on a daily basis to get rid of all the feeling, the assumption, and concretely make your decision on a concrete data.

ARTHUR: 13:28

Okay, so if I can summarize what you told me, uh, tell me if I'm a good learner. So we have to start with a clear goal. Actually, uh, we have to test things one by one, and the investment kind of will be worth it. And so, yeah, if you want to pick up fruits, uh, ensure you also have meaningful sample sizes also. Yeah, just learn, uh, refine and define. Yeah. That's it.

JÉRÉMY: 13:51

Totally agree.

ARTHUR: 13:52

And user centric decisions.

JÉRÉMY: 13:54

Absolutely. That's the main goal data.

ARTHUR: 13:57

Let's dive in a bit now about Odoo because it's also the subject of today. Odoo is also a tool that could support your a b testing. So how can Odoo support you through the process of AB testing? Can you tell me a bit about that?

JÉRÉMY: 14:10

Yes, Odoo makes actually the job pretty easier for you to AB testing. So I was speaking about the the fact that the marketing department can, can take a lot of time to create a duplicate campaign. So I don't know exactly how they work. So on a daily basis. But with Odoo it's so simple. You create the first version of an email, for instance, and then you just have to duplicate the the current email and that's it to have the B version. And you start working on the B version with all the same elements that you had in the first one. So you just set up the time. Yeah. The mailing list are integrated on Odoo, so you have all access to all the information about all other modules on the, on the system. And that's really powerful then because you just need to activate the two tests and make it running and it goes through automatically. You have the results. You even if you want, you can you can send it earlier. When you analyze and you say, okay, the campaign is, uh, everything is ready, let's go. And, uh, and Odoo makes the job all right.

ARTHUR: 15:10

And, and so you can automate and you can also manage your lead lists freely, like, uh, you can really organize your targets. How does it work?

JÉRÉMY: 15:18

Yes, there is a marketing automation tool on Odoo. But if we speak more in this case, yes, about the email marketing, what you can automate behind the lead list and everything that is that is already integrated is the fact that you can send automatically send the the best version based on the trigger that you set. If you say, okay, I have my two tests, the A and the B, and I want to send the the best mail in terms of open rate, for instance. So you make just the test with the two dates, be accurate in the timing and it's important side uh side advice on the AB testing. Because if you change the timing you send the first email or for example on a Sunday morning and, and the B version is, was on the Friday evening. Yeah. So that's not relevant. So send the same timing. And in terms of automation, if I come back to your question after it's just about yeah letting the system do so if after the test it's uh the system identify that it's the B version that has the best open rate, automatically add the scalable date for the final version, Odoo will send that one. And that's.

ARTHUR: 16:21

Magic. Is it some kind of I we use or do you know what's behind that?

JÉRÉMY: 16:25

Um, yeah. We can we can say that's not really a high, but uh, it's really based on, on metrics. So algorithms.

ARTHUR: 16:33

Yeah.

JÉRÉMY: 16:33

Let's say that you're just set up the fact that you want to send the final version on a specific date, and you first make the, the testing on a date earlier. And so if after this specific date, it's, uh, it's right. Noted. Because you have in real time the, the the measurement of your, of your campaigns in the system is after that, the system just on a factual basis, see that the open rate is higher on the B version at the scheduled date. It will take this condition into the consideration and send the right one. So not really high high, but uh, really. Uh, yeah. Action based on the data provided by the system.

ARTHUR: 17:10

To jump really quickly back to the LED list and just managing your lists, is it really easy to get your lists done in Odoo? Can you export, import, do whatever you want with it? Yeah.

JÉRÉMY: 17:21

It's, uh, easy as crazy because for me, that's even the the best feature. And the main reason why to to use the email marketing on Odoo, it's because the email marketing app is completely integrated with all the other apps around. You have the possibility to create mailing lists, make the segmentation with all the fields that you want to make. But what you have more is that if you want to pick up in the different modules around, for instance, you want to trigger all the customers that made the sales on the previous month and then purchase the specific products. So no matter if it was on the website directly sold by your sales team, for instance, or in, um, the point of sale solution, all those, uh, those information are available in real time on Odoo. So you are able to really make specific targeting based on all the data in real time on the system. And that's really powerful.

ARTHUR: 18:17

Okay. So finally, this tool is kind of a comprehensive understanding of our audience. It really lets us dive dive in.

JÉRÉMY: 18:25

Yeah, absolutely.

ARTHUR: 18:26

Do you have maybe a last advice for our audience?

JÉRÉMY: 18:29

Uh, yes. Uh, for who do choose the best and forget the rest. But regarding AB testing. Yes. Just be patient. Uh, avoid the conclusion and, uh, allocate enough time and sample to make reliable results in your a b testing. And if you are looking for a wonderful new challenge in sales and marketing, apply now on odoo.com/jobs. You got.

ARTHUR: 18:49

The tips. Thanks for joining us.

JÉRÉMY: 18:51

Thank you Arthur. See you.

ARTHUR: 18:53

Well that's a wrap for today. I hope you had a good time being with us. I look forward to catching you on our next episode. I invite you to the new pad section we launched after our first anniversary. Whew! That was so nice! Feel free to post questions. There will be more than happy to interact with you guys. Keep Odooing it. Bisous