JNACSISSN:2582-3817

A Novel Unsupervised Framework for DDoS Attack Detection Using Adaptive HDBSCAN and Ensemble-Based Feature Selection

Abstract

A Distributed Denial of Service (DDoS) attack floods a target system, server, or network with massive traffic from numerous sources, overwhelming it and disrupting normal functionality. Such attacks often leverage botnets—large groups of compromised devices—that generate malicious traffic at scale, making detection increasingly complex. This results in service unavailability for legitimate users, causing downtime, damage, or the exploitation of weaknesses. Existing models face challenges, including overfitting to training data, which limits their ability to generalize, and underfitting, which leads to missing critical attack patterns. Moreover, conventional detection methods struggle to adapt when attackers continuously evolve their strategies, leading to reduced robustness in real-world deployment. To overcome these challenges, this research paper presents an Adaptive Hierarchical Density-Based Spatial Clustering of Applications With Noise Approach (Ada- HDBSCAN) for detecting DDoS attacks using clustering. The process begins by simulating a cloud model and collecting input log data from the NSL-KDD dataset. Feature selection is performed using Gini Impurity-based Weighted Random Forest (GIWRF) and Sequential Forward Search (SFS) techniques to identify the most relevant features from the input data. This hybrid feature selection strategy ensures that redundant or noisy attributes are removed, thereby improving both efficiency and accuracy. Subsequently, clustering-based detection is carried out using Ada-HDBSCAN, which classifies the data as either DDoS-affected or normal. Ada-HDBSCAN is an innovative approach developed by adaptively modifying the clustering parameters of Hierarchical Density-Based Spatial Clustering of Applications With Noise (HDBSCAN). The developed Ada- HDBSCAN method achieved an Adjusted Rand Index (ARI) of 0.966, a V-measure of 0.959, and a Silhouette score of 0.978, respectively.

References

  • A. A. Soofi, M. I. Khan and F.-e.-. Amin, “A review on data security in cloud computing,” International Journal of Computer Applications, vol. 96, no. 2, pp. 95-96, 2017.
  • Z. Du, W. Jiang, C. Tian, X. Rong and Y. She, “Blockchain-based authentication protocol design from a cloud computing perspective,” Electronics, vol. 12, no. 9, p. 2140, 2023.
    1. M. Ali, S. U. Khan and A. V. Vasilakos, “Security in cloud computing: Opportunities and challenges,” Information Sciences, vol. 305, pp. 357-383, 2015.
    2. O. Osanaiye, H. Cai, K.-K. R. Choo, A. Dehghantanha, Z. Xu and M. Dlodlo, “Ensemble-based multi-filter feature selection method for DDoS detection in cloud computing,” EURASIP Journal on Wireless Communications and Networking, vol. 2016, no. 1, p. 130, 2016.
    3. W. Wang, X. Du and N. Wang, “Building a cloud IDS using an efficient feature selection method and SVM,” IEEE Access, vol. 7, pp. 1345-1354, 2018.
    4. M. Aamir, S. Mustafa and A. Zaidi, “Clustering based semi-supervised machine learning for DDoS attack classification,” Journal of King Saud University-Computer and Information Sciences, vol. 33, no. 4, pp. 436-446, 2021.
    5. W. Bhaya and M. EbadyManaa, “DDoS attack detection approach using an efficient cluster analysis in large data scale,” in Annual Conference on New Trends in Information & Communications Technology Applications (NTICT), 2017.
    6. N. S. Kharbanda, “Comparative Review of Supervised vs. Unsupervised Learning in Cloud Security Applications,” 2024.
    7. K. Golalipour, E. Akbari, S. S. Hamidi, M. Lee and R. Enayatifar, “From clustering to clustering ensemble selection: A review,” Engineering Applications of Artificial Intelligence, vol. 104, p. 104388, 2021.
    8. S. Dasari and R. Kaluri, “An effective classification of DDoS attacks in a distributed network by adopting hierarchical machine learning and hyperparameters optimization techniques,” IEEE Access, vol. 12, pp. 10834-10845, 2024.
    9. V. Q. Nguyen, V. H. Nguyen, L. T. Ngo and L. M. Nguyen, “Variational Deep Clustering approaches for anomaly-based cyber-attack detection,” Journal of Network and Computer Applications, p. 104182, 2025.
    10. T.-L. Nguyen, H. Kao, T.-T. Nguyen, M.-F. Horng and C.-S. Shieh, “Unknown DDoS Attack Detection with Fuzzy C-Means Clustering and Spatial Location Constraint Prototype Loss,” Computers, Materials & Continua, vol. 78, no. 2, 2024.
    11. V. Q. Nguyen, L. T. Ngo, L. M. Nguyen, V. H. Nguyen and N. Shone, “Deep clustering hierarchical latent representation for anomaly-based cyber-attack detection,” Knowledge-Based Systems, vol. 301, p. 112366, 2024.
    12. A. John, I. F. B. Isnin, S. H. H. Madni and M. Faheem, “Cluster-based wireless sensor network framework for denial-of-service attack detection based on variable selection ensemble machine learning algorithms,” Intelligent Systems with Applications, vol. 22, p. 200381, 2024.
    13. “NSL-KDD dataset,” 2025. [Online]. Available: https://www.kaggle.com/datasets/hassan06/nslkdd. [Accessed August 2025].
    14. R. A. Disha and S. Waheed, “Performance analysis of machine learning models for intrusion detection system using Gini Impurity-based Weighted Random Forest (GIWRF) feature selection technique,” Cybersecurity, vol. 5, no. 1, p. 1, 2022.
    15. S. Bashir, I. U. Khattak, A. Khan, F. H. Khan, A. Gani and M. Shiraz, “A novel feature selection method for classification of medical data using filters, wrappers, and embedded approaches,” Complexity, vol. 2022, no. 1, p. 8190814, 2022.
    16. X. Wu, D. Wang, M. Yang and C. Liang, “CEEMDAN-SE-HDBSCAN-VMD-TCN-BiGRU: A two-stage decomposition-based parallel model for multi-altitude ultra-short-term wind speed forecasting,” Energy, p. 136660, 2025.
    17. H. Sheng, Z. Huang, L. Ke, J. Zhang, Z. Zeng, M. Yasir and S. Liu, “Multi-scale vessel trajectory clustering: An adaptive DBSCAN method for maritime areas of diverse extents,” Ocean Engineering, vol. 334, p. 121461, 2025.
    18. F. J. Abdullayeva, “Distributed denial of service attack detection in E-government cloud via data clustering,” Array, vol. 15, p. 100229, 2022.