Works

Cact-US

Deliverables: infographic shown on this page

Background

Does the position of a checkout button affect a webpage's performance? I aimed to find out by conducting an A/B test on a cactus shopping site, and analyzing results through hypothesis testing. First, I created two versions of the shopping home page shown below with HTML/CSS/JavaScript (template provided by the CS1300 Course TAs).

As shown above, I moved the checkout button (in the form of a red shopping cart icon) to very bottom for page A, and very top for page B.


Evaluation Metrics

To test which version performed better, I ran statistical tests for these two metrics:
1. Time to Completion. In this case it refers to the time it takes to complete the desired task, which is to order $150 of cacti.
2. Return Rate. In this case, it refers to whether users have returned to the cacti shopping version (A or B) from the cacti checkout version.
I developed a set of null & alternate hypothesis regarding how my A/B design, which is placing the checkout button at the very bottom vs very top of the page, will affect each metric.

Hypothesis
Data Collection

I collected data from 38 user sessions, which took place during the course's class time. I sent my Heroku link that randomly selects interface A or B for users to complete. Regardless of the version they received, their task was to fill their cart with at least $150 of cacti/succulents. (Each person in the class did the same for each other, so everyone ended up clicking ~40 websites)

Data Cleaning

My dataset consists of several special cases:

  • Two users did not checkout
  • Two users did not meet the $150 requirement
  • Two users visited both A and B pages.
  • One user simply loaded the page and left

I omitted the one user who simply loaded the page and left since this added no value. However, I included all other cases above since they might yield interesting information as well as add more data points. Moreover, even for users who did not checkout or meet the requirement, the time they spent on both pages were reasonable enough.

Statistical Testing

Using Python & Scipy, I wrote a script that performed a t-test on my hypothesis on time to completion, and chi-squared test for my hypothesis on return rate. I then created the inforgraphic below that illustrates my process and results:

Limitations

Of course, I recgonize that there are various validity issues and confounding factors affecting my results & overall A/B test. Some of them include:

Takeaways

Through this project, I was able to put design principles in practice as well as conduct statistical analysis. I aimed to investigate effects of position of a checkout button to a webpage's performance, based on my belief that a higher-positioned checkout button would allow users to complete the task in a shorter amount of time but contain less affordances in that users are more likely to accidentally click on them and thus having to return to the page more frequently. (and vice versa for lower-positioned buttons).

After statistical analysis (at a=0.05), however, I concluded that there are no significant differences between the two metrics - time to completion and return rate - for the two versions. Of course, there are various underlying biases that may have affected the results. However, this also revealed that in terms of design principles such as affordances, navigation, and layout, the position of a checkout button may not be as significant as I perceived. Indeed, it did make a difference numerically when examing properties of my data such as averages and counts. However, it might also be that users are more focused on the other contents of the page, in this case the cacti, and that the precense of a checkout button, regardless of position, is sufficient to enable them to complete the given task at similar rates. In general, I believe that further testing is needed to confirm the effects of position of a checkout button to a webpage's performance.