Recently, deep learning has been widely utilized for object tracking tasks. However, deep learning encounters limits in tasks such as Autonomous Aerial Refueling (AAR), where the target object can vary substantially in size, requiring high-precision real-time performance in embedded systems. This paper presents a novel embedded adaptiveness single-object tracking framework based on an improved YOLOv4 detection approach and an n-fold Bernoulli probability theorem. First, an Asymmetric Convolutional Network (ACNet) and dense blocks are combined with the YOLOv4 architecture to detect small objects with high precision when similar objects are in the background. The prior object information, such as its location in the previous frame and its speed, is utilized to adaptively track objects of various sizes. Moreover, based on the n-fold Bernoulli probability theorem, we develop a filter that uses statistical laws to reduce the false positive rate of object tracking. To evaluate the efficiency of our algorithm, a new AAR dataset is collected, and extensive AAR detection and tracking experiments are performed. The results demonstrate that our improved detection algorithm is better than the original YOLOv4 algorithm on small and similar object detection tasks; the object tracking algorithm is better than state-of-the-art object tracking algorithms on refueling drogue tracking tasks.