Object detection and tracking are vital components of computer vision, with applications ranging in fields such as autonomous vehicles, surveillance, robotics, and augmented reality. Object detection focuses on recognizing and pinpointing objects within images or video, while tracking is responsible for following their movement through sequential frames. Deep learning advancements have significantly improved the performance and accuracy of these tasks. However, obstacles like changing object appearances, and occlusion in real-time systems continue to pose challenges. This paper proposes a novel Object detection and tracking system to analyze visual data and provide accurate predictions. The proposed work adopts an Eigen Value Decomposition-based Canny Edge Detection (EVD-CED) approach that acquires the image edges; conversely, a Spherical object detection technique is employed that extracts the position of the image. Further, the image edges and image position are combined, then the fused image is generated to detect the object. Further, an innovative model called LeNet architecture recognizes the particular position of the moving target. Once the position of the object is detected, the object tracking process is carried out using the Improved Extended Kalman Filter-based Kernel Correlation Filter Tracking (IEKF- KCF) scheme. Thereby, the proposed work accurately detects the object and performs the tracking process efficiently. The IEKF-KCF attained the IOU of 0.854, NIOU of 0.827, MOTP of 0.844, and MAP of 0.814, respectively.