Text Classification on SparkDistributed MLlib over 13,000 features
Four algorithms, one dataset, and a reminder that an aggregate accuracy figure can hide a class the model essentially cannot do.
The problem
Distributed text classification across 27,528 training and 7,030 test rows with 13,003 features, tuned to run inside 45GB of a 51GB Colab machine without spilling.
The approach
A PySpark MLlib pipeline — tokenisation, TF-IDF vectorisation, and a cross-validated comparison of Logistic Regression, Naive Bayes, Random Forest and Decision Tree — with Spark configured explicitly for the hardware: Kryo serialisation, adaptive query execution, skew-join handling and partition counts matched to core count. Then a per-category confusion analysis rather than a single score.
The outcome
Logistic Regression won at 92.76% accuracy and 0.9273 F1, just ahead of Naive Bayes at 92.66%; the tree models sat at 73.07%. The useful finding was the spread underneath: per-category accuracy ranged from 99.79% down to 29.22%, so the headline number was carried entirely by the easy classes. Graded 90.