Preliminary Comparative Analysis of GPU and NPU Inference Stability for Battery Defect Inspection under Continuous Operation

As demand for lithium-ion batteries for electric vehicles (EVs) surges, requirements for efficiency and stability in battery production are also growing. Accordingly, research on automating battery defect detection with AI-based vision has been actively pursued [1, 2]. However, production lines generally operate 24 hours a day without interruption, and other programs such as logging and communication run concurrently on the same system. Therefore, inference latency and its variation under sustained operation may affect the timeliness of inspection results. In continuously operating systems, tail latency can also be relevant to meeting real-time requirements [3, 4]. However, prior visual inspection studies have mainly focused on detection accuracy and average inference speed on a single GPU, and have not sufficiently addressed inference stability under long-term operation and load [1, 2]. In this study, we therefore compare and analyze the latency characteristics and accuracy retention of an industrial vision detection model under long-term operation on a GPU and on an NPU, which belongs to the class of dedicated accelerators [5] whose deterministic execution has been reported to be advantageous for tail-latency requirements. The contributions of this study are as follows. • We deploy the same COCO-pretrained detection model (YOLO11x) on an NPU (NXP Ara240) and a GPU (RTX 5080), and measure latency characteristics over 150 minutes of continuous operation in an environment where both devices share a single host and perform inference concurrently. • We compare the two devices using not only the average latency but also tail-latency and consistency metrics such as p99, standard deviation, and coefficient of variation. The GPU was about 5.9× faster, whereas the NPU showed smaller latency variation and a smaller tail latency relative to the mean (the difference between p99 and the mean). • We examine whether detection results and accuracy (mAP) are maintained during repeated operation, thereby evaluating not only speed but also the consistency of inspection quality under long-term operation.

A. Battery Visual Inspection Deep learning–based visual inspection has been widely used for defect detection in battery manufacturing. Kang et al. [1] detected pouch-cell surface defects with a lightweight YOLOv11n and reported an average inference time of 9.43 ms on an A100 GPU. Xu et al. [2] proposed a network for detecting fine surface defects on prismatic batteries and applied it to an automated optical inspection (AOI) system on an actual production line, but reported that the end-to-end pipeline reached only about 19 FPS and did not meet strict real-time requirements. As such, prior studies have mainly reported detection accuracy and average inference speed on a single GPU, and have not addressed latency stability under long-term operation or when sharing resources with other programs. B. Sources of Inference Latency Variation Inference performance can vary with operating conditions. Benoit-Cattin et al. [6] showed that running CNNs for long periods on a CPU-based edge device significantly degrades throughput due to thermal throttling. Gong et al. [7] reported that on an edge GPU (Jetson Orin Nano), heat-induced changes in operating frequency cause large fluctuations in detector inference latency; for Faster R-CNN, the mean was 434.6 ms with a standard deviation of 139.8 ms. Even for high-performance GPUs, Sinha et al. [8] showed that GPUs of the same model differ in performance by 8% on average and by up to 22% due to power and thermal management. Dean and Barroso [4] identified shared-resource contention, background processes, and power limits as major causes of tail latency, and MLPerf Inference [3] likewise specifies that inference latency be evaluated by percentiles rather than by the mean. C. Dedicated Accelerators and Latency Predictability Dedicated neural network accelerators have attracted attention for their predictable latency. Jouppi et al. [5] reported that the deterministic execution model of the TPU is a better match to 99th-percentile response-time requirements than the time-varying optimizations of general-purpose processors, such as caches and multithreading. More recently, Tummalapalli et al. [9] showed that under sustained load an edge NPU has lower throughput than a GPU but almost no performance variation; however, their target was LLMs and their measurements were limited to 20 iterations. Therefore, studies that jointly compare the latency distribution and accuracy retention of discrete NPUs and GPUs under long-term operation for industrial vision detection models are still lacking.

Table 1 summarizes the host and the accelerators. All experiments were conducted on the same host, which consists of an AMD Ryzen 9 9950X CPU (16 cores/32 threads), about 60 GB of memory, and a Linux operating system. The accelerators compared are an NXP Ara240 discrete NPU (M.2 module, up to 40 eTOPS, 16 GB LPDDR4) and an NVIDIA GeForce RTX 5080 GPU (16 GB GDDR7, 360 W power limit). The GPU was run with PyTorch, and the NPU with the Ara runtime r3.0. The models currently provided by NXP for the NPU are limited. We therefore used Ultralytics YOLO11x [10], a model of the YOLO family [1] that has been used for battery defect inspection in industrial settings. YOLO11x is an object detection model pretrained on the COCO 2017 training set [11] (56.9M parameters, 195.3 GFLOPs, official COCO mAP50–95 of 54.7%), and we used the public weights as-is without additional training. For the NPU, we used the model that NXP quantized to INT8, compiled for the Ara240, and released [12]; for the GPU, we used the original Ultralytics weights. NXP reports an accuracy of 51.94% for the NPU INT8 model (54.7% in FP32). Because these official values are evaluated including low-confidence detections, they cannot be directly compared with the mAP in this study, which is evaluated only on detections with confidence ≥ 0.25. The input resolution was fixed at 640×640 and the batch size at 1, and post-processing was applied identically on both devices with a confidence threshold of 0.25 and an NMS IoU of 0.7. Accordingly, we used the 5,000 images of COCO val2017 as evaluation data. COCO val2017 is the official evaluation set that was not used for model training, and its ground-truth annotations allow detection accuracy to be checked during operation. The images were fed repeatedly in a fixed order for 2 hours and 30 minutes. The two devices performed inference concurrently on the same host. This setup approximates a resource-sharing environment in which multiple inspection programs may use the same industrial PC. Each device started the main measurement after 20 warm-up inferences, and for each image, we recorded the end-to-end latency (from image input to the completion of post-processing) and the device execution time (backend). For GPU measurements, synchronization was performed before and after each measurement to eliminate the effect of asynchronous execution. Device temperatures were also recorded at 1-second intervals. The experiment was conducted twice independently: Experiment 1 evaluated latency stability over 150 minutes (9,000 s) of continuous operation, and Experiment 2 ran for 40 minutes (2,400 s) to confirm the reproducibility of Experiment 1 and to save detection results for accuracy evaluation. Latency was evaluated using the mean together with p50, p95, p99, p99.9, the maximum, the standard deviation, and the coefficient of variation (CV = standard deviation/mean), because tail latency and variability, in addition to the mean, may affect whether a continuously operating system meets real-time requirements [3, 4]. Changes over time were assessed by comparing the mean latency of the first and last approximately 5 minutes. Accuracy was computed as mAP50–95, mAP50, and mAP75 using the official pycocotools COCO bbox evaluation, and we verified that the detection results were identical across all dataset repetition cycles.

Table 2 shows the results of the 150-minute concurrent run. The mean end-to-end latency of the GPU was 11.37 ms (87.86 FPS), about 5.87× faster than the NPU's 66.74 ms (14.98 FPS). In contrast, the NPU showed greater consistency in latency. The CV was 0.81% for the NPU and 6.46% for the GPU, and the standard deviation, an absolute measure of variation, was also smaller for the NPU (0.539 ms) than for the GPU (0.734 ms). The tail latency relative to the mean, i.e., the difference between p99 and the mean, was likewise smaller for the NPU at 1.91 ms (2.9% of the mean) than for the GPU at 2.46 ms (21.6% of the mean). However, the absolute p99 was lower for the GPU, at 13.83 ms versus 68.65 ms for the NPU. Neither device had any execution failures during the 150 minutes. Over the 150 minutes, the mean latency of the first and last approximately 5 minutes changed from 11.335 ms to 11.372 ms (+0.32%) for the GPU and from 66.706 ms to 66.731 ms (+0.04%) for the NPU, showing no noticeable latency increase on either device during the observation period. The p99 of each 5-minute interval also remained stable, within 13.77–13.87 ms for the GPU and 68.39–68.90 ms for the NPU. In terms of the latency distribution, the NPU was tightly concentrated around its mean, whereas the GPU showed a gentle shoulder to the right of its main distribution (around 13 ms) (Figure. 1). This difference may result from a combination of per-image workload, post-processing, and device execution characteristics, and will be addressed in future work. In the separate 40-minute experiment, the mean latency (GPU 11.38 ms, NPU 66.70 ms), p99 (13.82 ms, 68.58 ms), and CV (6.46%, 0.85%) were nearly identical to those in the 150-minute experiment, indicating similar results across the two runs. However, the maximum NPU latency was 109.23 ms, which was higher than in the 150-minute experiment (74.99 ms). This was a rare delay not accompanied by any increase in the mean or p99, and its cause likewise remains to be determined in future work. Table 3 shows the COCO bbox accuracy for the detection results from the 40-minute experiment. The mAP50–95 was 48.16% for the GPU and 46.58% for the NPU, 1.58 percentage points lower for the NPU. However, no change in detection results or accuracy was observed across all complete dataset repetition cycles (41 for the GPU and 7 for the NPU).

In this study, the two accelerators showed a clear difference in speed and latency consistency. The GPU delivered about 5.87× higher processing performance than the NPU, whereas the NPU showed smaller values not only in CV (0.81% vs. 6.46%) but also in the standard deviation, an absolute measure of variation (0.539 ms vs. 0.734 ms), and in the difference between p99 and the mean (1.91 ms vs. 2.46 ms). That is, the lower variation of the NPU is not merely a relative effect of its larger mean latency. However, since the absolute p99 of the GPU (13.83 ms) is lower than that of the NPU (68.65 ms), the advantage of the NPU is limited to a shorter tail relative to its mean. Meanwhile, neither device showed an increase in the mean or per-interval p99 latency over the 150 minutes, and no thermal performance degradation appeared within the observed range. Therefore, the tail latency observed on the GPU (the distribution around 13 ms in Figure. 1) may stem from per-image workload, post-processing, and device execution characteristics rather than from thermal factors caused by long-term operation. From an industrial inspection perspective, these results show that latency consistency and absolute speed must be considered together. When the 25 FPS (40 ms) per-line criterion used in prior work [1] is applied, all GPU frames met the criterion with a maximum latency of 18.99 ms, whereas all NPU frames with YOLO11x exceeded it, with a minimum latency of 65.10 ms. Therefore, low-latency variation alone does not make the NPU more suitable for industrial inspection; latency consistency becomes meaningful only once the required time is met. Nevertheless, the relatively narrow NPU latency distribution may be relevant to deadline analysis if a lightweight model meets the required processing time. However, given the rare maximum latency of 109.23 ms observed during the 40-minute experiment, worst-case predictability cannot be established from these measurements. In terms of accuracy, the mAP50–95 measured on the NPU execution path was 1.58 percentage points lower than on the GPU, but the detection results of both devices were identical in every repetition cycle over the 40 minutes, and no change in inspection quality over operating time was observed. This study has the following limitations. First, because the two devices ran concurrently on a single host, the effect of load relative to standalone execution was not evaluated. Second, the operating time was 150 minutes, and stability under actual 24-hour continuous operation requires further verification. Third, the NPU used a vendor-compiled INT8 model, whereas the GPU used the original PyTorch weights, so the deployment methods and precision differ. Fourth, NXP states that COCO val2017 was used for the INT8 calibration of the NPU model [12], so the calibration images may partially overlap with our evaluation data. Finally, an actual battery defect detection model could not be deployed due to compatibility issues during NPU compilation, so a general-purpose COCO-pretrained model was evaluated instead. In future work, we plan to extend these results by comparing with standalone execution, long-term operation lasting 24 hours or more, same-precision comparisons, and NPU deployment of a defect detection model.