Supervised Learning Techniques for E-Commerce Orders¶

This phase applies supervised learning techniques to the cleaned e-commerce dataset to predict the payment type of each order

The primary objectives are:

1. Preprocess the dataset by appropriately handling numerical and categorical features to prepare it for model training.
2. Compare the performance of 3 supervised learning techniques (Logistic Regression, Decision Tree & Random Forest) based on standard evaluation metrics, class imbalance considerations, model complexity, and practical deployability.
3. Using these comparisons, we will recommend the ideal model to be utilised by the E-Commerce firm to support its business objectives.

Setup¶

The environment set-up utilises pandas and numpy for data manipulation, matplotlib and seaborn for visualisation and sklearn for preprocessing, model training, and evaluation.

Part 1: Data Preparation¶

We will first prepare the transformed dataset so that it can be used for supervised classification. Proper data preparation is a critical step in the machine learning pipeline, as model performance and validity depend heavily on the quality, structure, and representation of the input data.

Data Loading and Splitting¶

In this section, we use the processed dataset ecommerce_orders_cleaned.csv from Assignment 1. To predict the payment type of an order, we set payment_type as the target variable, while all remaining columns are treated as input features. We then partition the dataset into training and test sets using train_test_split() with a test size of 0.2 and the pre-defined RANDOM_STATE to ensure reproducibility.

It is important to have a test set as it is used to evaluate how well the trained model performs against a completely new set of data which is previously unseen. This helps to detect overfitting and provides a fair comparison of the model performance.

Handling Categorical Variables¶

Categorical data has to be processed before it can be used for machine learning because the models work with numbers only and not string objects. We do so by identifying all categorical columns in the training data first and then apply label encoding to convert category labels into numerical representations so that the models can interpret the categories.

Encoders are fitted on the training data only and subsequently used to transform both training and test sets. This can help to prevent data leakage and ensure consistency throughout the work flow. Any unseen categories are handled by having a placeholder label unknown_category to prevent transformation errors and improve model robustness.

Handling Numerical Variables¶

We handle the numerical variables by performing feature scaling. StandardScaler from scikit-learn is used to perform this scaling which standardises the numerical features by converting the mean and standard deviation to 0 and 1 respectively. This allows the numerical features to be on a common scale.

This is important because it prevents feature dominance by minimising the effect of different units of measurement of each feature. For example from X.describe() below, we can see in the original dataset that order_total_value ranges from 6.12 to 2890.55 while num_items only ranges from 1 to 8. As such, when it comes to distance-based models, order_total_value will dominate num_items as a feature, resulting in the model being biased towards the former.

Part 2: Applying Machine Learning Models¶

In this section, we train and evaluate 3 supervised classification models to predict the payment method of an order. We then conduct a systematic comparison of their predictive performance and practical suitability for deployment.

Logistic Regression¶

Logistic Regression is a linear classification model that models the log-odds of the target class as a linear combination of the input features.

Decision Trees¶

Decision Trees is a tree-structured model that makes predictions by recursively splitting the feature space based on feature values.

Random Forest¶

Random Forest is an ensemble learning model that makes predictions by combining multiple decision trees built on different subsets of data and features. By using aggregated voting, it reduces overfitting and improves model performance.

Part 3: Evaluation & Visualization¶

We evaluate and comapre the performance of all three models using standard evaluation metrics and confusion matrix visualisations for a comprehensive and robust analysis.

1. Evaluation metrics:

   • Accuracy
     • Measures the overall proportion of correct predictions which provides a general performance overview
     • Accuracy = (TP + TN)/(TP + TN + FP + FN)
   • Precision
     • Measures how many predicted positives are actually correct
     • A high precision implies low false positive errors.
     • Precision = TP/(TP+ FP)
   • Recall
     • Measures how many actual positives are correctly identified
     • A high recall implies low false negative erros
     • Recall = TP/(TP + FN)
   • F1-score
     • Harmonic mean of precision and recall
     • F1 = (2 x Precision x Recall)/(Precison + Recall)

2. Confusion matrix:

   • The confusion matrix are presented as heatmap visualisations to illustrate the distribution of correct and incorrect predictions across classes.
   • This helps us to identify misclassification patterns and class-level performance differences.

Based on the results above, we discuss the strengths and weakness of each model, as well as how class imbalance is being handled.

Logistic Regression

• Strengths:
  • Easy to implement
  • Highly interpretable - The coefficient of each feature describe the odds ratio of each feature, providing an intuitive understanding of which feature drives the model prediction
• Weaknesses:
  • Prone to poor performance when there are non-linear relationship between the features and the dependent variable, which is highly likely in real-life scenarios involving data with complex patterns
• Class imbalance handling:
  • Can adjust threshold to treat lower probabilities as actionable with considerations given to business risk tolerance
  • Can apply resampling first (downsample of majority class or oversample minority class)

Decision Tree

• Strengths:
  • Able to capture non-linear relationships
  • Easy to visualise (for smaller trees)
  • Does not require scaling
• Weaknesses:
  • Difficult to interpret large and complex trees with many leaves
  • Highly prone to overfitting. A deep tree might "memorize" specific orders in the training set rather than learning general patterns, leading to poor performance on the test set.
  • High variance whereby small changes in the training data can lead to very different tree structures and predictions, hence, it is not suitable for real-life production pipelines
• Class imbalance handling:
  • Can apply the class_weight parameter to penalise minority class errors more heavily
  • Can apply resampling first as well

Random Forest

• Strengths:
  • Able to capture non-linear relationships
  • Lower variance and overfitting compared to decision trees as the predictions are averaged across multiple trees
• Weaknesses:
  • Difficult to interpret decision logic since there could be hundreds of trees
  • High computational cost due to the training of multiple trees
• Class imbalance handling
  • Similar to Decision Tree, a class_weight parameter can be applied to each bootstrap sample (subset of the data for each tree)

Part 4: Model Selection & Final Recommendation¶

Based on the evaluation and comparison discussed above, we provide our recommendation for the model to be selected

Justifications¶

The model selection is based on the following justifications.

1. Evaluation Metrics:

   • Random Forest outperformed the other models in all the metrics as shown above, in particular, it has the best F1 score of 0.57.
   • Due to the class imbalance (credit_card: 60%, voucher: 30%, points and debit card <= 5%), accuracy is not a suitable evaluation metric. Meanwhile, there is no strong preference towards false positive errors or false negative errors given that we are simply predicting payment types. Hence, F1 score is the most suitable evaluation metric as it combines both Precision and Recall.

2. Class Imbalance Considerations:

   • Random forest handles class imbalance in a more robust manner due to bootstrap sampling and the ensemble diversity of multiple trees.
   • Logistic regression only learns a single global deicsion boundary which fails to capture small local pockets of the minority class. A decision tree has high variance and may overfit for the majority class.
   • On the other hand, the bootstrap sampling in random forest allows minority signals to be learned repeatedly and the ensemble helps to stabilise minority detection.

3. Model Complexity and Interpretability:

   • Logistic regression is easy to implement and intuitive to understanding using the feature coefficients. Decision tree can still have interpretable decision logic if the tree is shallow.
   • A random forest is the most complex and has the lowest interpretability among the 3 models. However, alternative methods such as SHAP can be used to understand how each feature drives the prediction.

4. Practical Deployment Considerations:

   • Logistic regression is the most simple to implement and it runs quickly. Decision tree runs quickly too but its instability can cause maintenance issues in a production pipeline.
   • Random forest requires a high computational cost due to its ensemble nature of handling multiple trees and high memory usage.

Why this model is preferred over others: Random Forest is preferred over Logistic Regression and Decision Trees because it provides stronger predictive performance, greater stability, and better robustness to class imbalance. By combining many decision trees trained on different bootstrap samples, the model captures complex nonlinear patterns while reducing the variance and overfitting risk of a single tree. Although more complex, interpretability can be maintained using SHAP values, which explain how each feature contributes to predictions. With modern production systems capable of handling large-scale computation, the additional training cost is generally outweighed by the model's improved accuracy and reliability, making Random Forest a strong overall modeling choice.

Model Explainability/Interpretability¶

Lastly, we compare the explainability and interpretability of the models

Logistic Regression:

• Logistic Regression has the highest interpretability because it is easiest to explain mathematically. This is because of the coefficients of each feature which provide a magnitude and direction that can be used to assess the feature's impact on the prediction.

Decision Tree:

• Decision Tree can have high interpretability if the tree is shallow. This is so because the decision logic is essentially a sequence of if-else logic which can be traced. However, the interpretability decreases if the tree becomes too deep or have too many leaves.

Random Forest:

• Being an ensemble of multiple decision trees, Random Forest is not directly interpretable. Instead, tools such as SHAP can be applied to help us understand feature importance.

Part 5: Conclusion¶

• This assignment provided provided us with a hands-on experience of processing a dataset and subsequently train the processed data using different models to evaluate which model provides the best performance.
• Among the models tested, we found that Random Forest achieved the strongest predictive performance and demonstrated greater model stability, largely due to its ensemble structure that reduces variance compared to a single decision tree. It also captures nonlinear relationships that logistic regression may not model effectively. Although Random Forest is more complex, its interpretability can still be supported through model explanation tools such as SHAP, which help quantify the contribution of individual features to predictions.
• Key takeaways:
  • Firstly, model selection should not rely solely on predictive performance. Other factors such as interpretability, computational complexity and deployment feasibility must also be considered depending on the application context. This is particularly important in a business setting where resources and timeline are important considerations to factor into the development of machine learning pipelines.
  • Another area to highlight would be how class imbalances are being handled by the different models. In many real-life scenarios (customer churn, fraud detection, medical diagnosis), class imbalance scenarios can be common. Appropriate techniques such as class weighting, resampling, or threshold adjustment should therefore be incorporated into the modeling process to ensure fair and meaningful evaluation.
  • Lastly, we have to be mindful of how we pre-process the data before using them as features in a model. In this assignment, we touched upon Label Encoder to encode categorical columns, and Standard Scaler to normalise numerical columns. However, there are other techniques such as One-hot encoding or ordinal encoding, min-max scaling (for distance-based models) and robust scaling (to handle outliers), which may be more suitable depending on feature characteristics and the choice of model.

Overall, this assignment showcased to us the development of an end-to-end machine learning workflow, from preprocessing and model training to evaluation and practical deployment considerations. It prompted us to consider how an effective model workflow requires balancing predictive accuracy with interpretability, and operational feasibility.