10 Experiments Ideas

Introduction:

The Experiments feature allows you to split-test your already existing business operations and new business decisions.

➡️ Click here to read our guide on how to create, run and understand Experiments.

 

This guide lists possible experiments you can run in your e-commerce business to uncover insights, hidden profits, and unnecessary costs.

Making it easy for you to start experimenting today.

 

“Group sizes might vary, depending on the size of your customer base. For most accurate results, we recommend using at least 10% in each test group”

 

1. Candy in each order

Hypothesis: Candy in each order increases retention and LTV.

See if you can boost retention by offering candy or other free goods in orders.

Create two groups:

1. Control group (80%): Receives nothing
2. Group A (20%): Receives candy in each order

 

Export Group A audience and insert them into your warehouse management system.

Monitor both groups' LTV and see if adding candy in each order impact your retention strategy.

➡️ Unsure on how to monitor LTV within Experiments? Check out our understanding experiments article.

 

2. Is 1-day delivery worth it?

Hypothesis: 1-day delivery increases retention and LTV.

Changing business operations is often associated with new costs and time spent. Before jumping into something new, such as 1-day delivery, it is important to test if it impacts profit.

Create two groups: 

1. Control Group (80%): Receives the current delivery time
2. Group A (20%): Receives 1-day delivery

Export Group A audience and insert them into your warehouse management system.

See how the difference in delivery days affects their LTV.

 

3. Private-label products versus 3rd party products

Hypothesis: Private-label items generate more profits than 3rd party products.

 

For any e-commerce selling both private-label and 3rd party items, it is ideal to test how well your private-label items sell. Private-label products usually have a much higher profit margin versus 3rd party items.

Create two groups: 

1. Control Group (80%) - Keeps receiving the same newsletters with 3rd party products.
2. Group A (20%) - Receives newsletters with only private-label products.

Activate both groups into your email service provider and start sending them different newsletters.

💡 Tip: Understand the product insights page to claim a deeper knowledge of which product to use

 

 

4. How many newsletters should you send per week?

Hypothesis: Sending more newsletters per week will increase revenue.

 

Find the best email strategy by testing the sending frequency on your newsletters.

Create three groups:

1. Control Group (60%) - Keep receiving 1 newsletter per week
2. Group A (20%) - Receives 2 newsletters per week
3. Group B (20%) Receives 3 newsletters per week

Activate the three groups into your email service provider and start sending them newsletters.

 

5. Personalized notes in orders

Hypothesis: A personalized Thank-You message will boost LTV

 

Similar to adding candy to each order, placing a personalized handwritten thank you in each order might also be a way to boost LTV.

Create two groups:

1. Control Group (60%) - Doesn’t receive a note
2. Group A (40%) - Receives a handwritten note in their order

Export Group A audience and insert them into your warehouse management system. Start adding handwritten notes to this group's orders.

Monitor the two groups' LTV and see if you should change your retention strategy.

 

6. What happens if you turn off marketing?

Hypothesis: Turning off marketing will severely impact customers’ LTV.

 

Experimenting doesn’t always have to be about finding new growth opportunities.

It can also be used to validate your already existing business operations. Turning off marketing for a percentage of your current customer base can give you insight into just how much marketing affects customer LTV.

Create two groups:

1. Control group (80%) - Keeps receiving advertising
2. Group A (20%) - Exclude from all paid marketing

 

7. What product category drives the second purchase?

Hypothesis: There exists a product that drives more second purchases.

 

In this experiment, you need to start using the filter option.

To find out what drives second purchases, we first need all customers who only have made 1 purchase at your store.

In Experiment -> Add Filter - Add Total Orders = 1

You now want to create testing groups based on how many product categories you have. In this example, the business fashion e-commerce store.

1. Control group (40%) - Keep your current retention strategy
2. Group A (15%) - Enroll in email flow focusing on Pants
3. Group B (15%) - Enroll in email flow focusing on Jackets
4. Group C (15%) - Enroll in email flow focusing on Shirts
5. Group D (15%) - Enroll in email flow focusing on Shoes

Activate Groups A, B, C, and D into your email service provider and enroll them into the category-specific email flows.

 

“Best practice: Use product insights to see best-performing products in each category and use those products to build your email flow”

💡 Tip: Understand the product insights page to claim a deeper knowledge of which products are trending.

 

8. How price sensitive are your customers?

Hypothesis: Customers will buy more expensive products if put in front of them.

 

You want to send emails with products, ranging in price from average to high. This can help you find out if your customers are more willing to buy your expensive products if shown to them.

Create two groups:

1. Control group (80%) - Keep your current price strategy
2. Group A (20%) - Only send emails with products that have a high avg. sales price. (Can be found under Products)

Activate group A into your email service provider and enroll them in a flow with high avg. price products.

 

9. Does offering a one-time discount help repeat purchases for win-back customers?

Hypothesis: Win-back customers have a higher chance of repeat purchasing if offered a discount.

 

In Experiments -> Add Segment -> Win-back Customers

You want to find out if a discount code can make churning customers come back. Use the win-back customers segment and create two groups:

1. Control group (50%) - Doesn’t receive a discount code
2. Group A (50%) - Receives a discount code

Activate Group A into your email service provider and send them a discount code.

“Best practice: Take this experiment even further by making more test groups, and test different amounts of discount, to find the perfect x%”

 

💡 Tip: Consider using the machine learning filter 'probability of churn' as part of your experiment.

⏰ Time: Run the experiment for 3-6 months in order to have a substantial data foundation to base conclusions on.

10. Why do your most loyal customers keep coming back?

Hypothesis: Most Loyal Customers will return even if not advertised to.

 

Experiments are not only for uncovering hidden profits but also unnecessary costs. This Experiment helps highlight if your ad spend goes towards customers, who would’ve purchased in your store, regardless of your ads.

In Experiments -> Add Segment -> Most Valuable Customers

Create two groups:

1. Control Group (80%) - Keep current marketing strategy
2. Group A (20%) - Exclude them from all Facebook Advertising

 

Activate group A into your Facebook and exclude them from all advertising.

Monitor how Group A LTV fairs against the control group. Remember the ad spend that you save, by not advertising to them.

 

 

🙇🏽‍♂️ Know how we could improve? We would love some feedback!