Lite-TemporalViT: Decoupled Spatial-Temporal Transformer for Ultra-Low Latency Aerial Drone Surveillance

Autonomous micro aerial vehicles (MAVs) and small drones are increasingly used in crucial field tasks like disaster response, forest fire tracking, urban traffic moni-toring, perimeter security, and search-and-rescue [1]–[3]. To fly safely and independently through changing out-door environments, these drones need onboard vision systems that can read spatial details from images while tracking movement across multiple frames in real time [2]. Traditional edge perception setups analyze the video feeds frame-by-frame using the standard 2D Convolu-tional Neural Networks (CNNs) [4]–[6]. While the stand-ard 2D CNNs run more quickly and it requires the mini-mal memory, they cannot track the motion over time. Crucial motion details like the vehicle speeds, direction changes, temporary blockages, and object interactions across frames are completely lost when looking at single frames, which weakens the drone's situational awareness. To bring the motion tacking into the video vision sys-tems, researchers usually rely on the 3D Convolutional Neural Network (CNNs) [8], [9]. On the other hand, if we look into standard 2D CNNs they ran quickly and they require minimal memory but cannot track motion over-time. Crucial motion details like the vehicle speeds, di-rection changes, temporary blockages, and object inter-actions across its frames completely lost when looking at single frames, which basically weakens the drone's situa-tional awareness. When it placed on the low-power embedded hardware like the NVIDIA Jetson Orin Nano or ARM Cortex chips, these combined networks basically create the heavy memory bottlenecks [4]. When these large tensor opera-tions run they flood the onboard cache memory, which causes the processing delays, drains battery life quickly, and generates high heat that forces the system to slow down to prevent overheating. When drones fly close to the ground at high speeds, even the tiny processing delays can jeopradize the flight safety. If visual data takes too long to process, the flight controller receives delayed information, leaving less time to avoid obstacles and significantly increasing crash risks. That is why creating a lightweight video processing system that captures motion across frames in under two milliseconds while using minimal memory is so crucial for modern autonomous aircraft. To fix these computing bottlenecks, we designed Lite-TemporalViT, a factorized hybrid architecture built or designed specifically for low-power drone operations. We seperated the process of pulling spatial details out from temporal attention mechanism across multiple frames, which basically cuts the huge chunk of unneces-sary computational workload. Our model compresses 4-frame video clips into compact temporal sequences, pro-cessing clips in 1.58ms (±0.08 ms) on an NVIDIA T4 GPU (~632 clips/sec or ~2,528 frames/sec) while using only 1.28 million parameters. Problem Statement and Hardware Bottlenecks The hardware chips that are built into drom=nes run into strict memory limits and the heat constraints. High-end workstation GPUs get access to high-speed memory architectures, but small embedded boards have to rely on shared LPDDR memory with much narrower bus routes [4]. When you try to run 3D convolution filters or full space-time attention layers on these smaller boards, the system spends a massive amount of time bouncing data back and forth from memory, which becomes the main cause of processing lag. Lite-TemporalViT fixes this problem by breaking down and simplifying the spatial feature maps first before tracking motion across time steps, keeping memory transfer demands well within what the embedded cache can handle. Main Contributions The main contributions of this work are as follows: 1) Decoupled Architecture Design: We designed a system that seperates the 2D spatial image feature extraction from temporal motion tracking that ba-sically ensures the computational demands grow manageable and linear as the length of the clip in-creases. 2) Low Latency and High Speed: Our model achieves sub-2ms clip execution times (1.58 ms, corre-sponding to ~632 clips per second and ~2,528 ef-fective frames per second) using just 1.28 million parameters, outperforming models like 3D-ResNet18 [5], SlowFast-Lite [12] and ViViT-Tiny [14] in both throughput and accuracy. 3) Reproducible Hardware Testing: We perform rig-orous hardware benchmarking reporting latency variability across multiple hardware setups includ-ing the NVIDIA T4, Jetson Orin Nano, Jetson Nano 4G and Raspberry Pi 4 under synchronized CUDA streams and fixed power profiles. 4) Detailed Configuration Tests: We ran extensive tests to evaluate different model choices, analyz-ing clip buffer length network layer depths, float-ing-point levels and temporal attention maps.

A. Spatio-Temporal Convolutions and Factorization Early deep learning tecniques for the video classifica-tion built on the standard 2D CNNs into 3D by running 3D convolutional filters across sequences of the video frames (such as C3D [8], 3D-ResNet [5], and I3D [9]). Even though 3D convolutions also do a solid job catch-ing the spatial details and the motion at the same time, they basically requires a huge amountsof floating-point operations (FLOPs) and push Memory Access Costs (MAC) through the roof. To ease up on those heavy com-puting demands, factorized architectures like Pseudo-3D (P3D) [10] and R(2+1)D [11] broke 3D convolutions down into the separate 2D spatial filters and 1D temporal filters. SlowFast networks [12] went a step further by using a dual-pathway system to handle detailed spatial frames and rapid temporal motion at different frame rates. Still, despite shaving off a chunk of FLOPs, these 3D CNN backbones hit a wall with the memory limits, which leads to frequent performance stalls on low-power drone hardware. B. Video Vision Transformers Following the success of Vision Transformers (ViT) on the single static image processing [7], newer models like the TimeSformer [13], ViViT [14], and the Video Swin transformer [19] brought the top tier accuracy to the vid-eo analysis by applying self-attention across spatial and temporal visual tokens together. The main catch is that full space-time attention scales quadratically as the total number of the tokens grows. Even factorized models that then split up space and the time attention [17] still de-pends upon massive parameter counts frequently topping 10 million parameters which then quickly choke the memory bandwidth of the small edge hardware. Lite-TemporalViT sidesteps this bottleneck by flattening the spatial feature maps into vectors before it passed down to sequence model, ensuring that the computational de-mands scale strictly linearly as video frames increase. C. Edge Computing and Aerial Surveillance Constraints Deploying the computer vision models directly onto the autonomous drones means dealing with the tight boundaries around power draw, thermal limits, and the memory capacity [3], [4]. Since drones relies entirely on the on-board battery reserves, the central processing hardware then has to share the power with the electric motors that helps drone airborne. When the heavy algo-rithms run, they produce excess heat that forces the em-bedded chip to lower its clock speed to basically avoid the permanent hardware damage. Beyond these physical constraints, real-world aerial video streams routinely deal with unpredictable motion jitter, camera vibration, shifting lighting conditions, and drastic size changes of objects in the frame [2]. Existing video vision frame-works simply cannot deliver the high accuracy, sub-2ms latency, and minimal VRAM memory footprint needed to make the real-time autonomous flight safe and reliable. D. Overview of Existing Paradigms Current research shows a clear gap between heavy spa-tiotemporal transformers built for high-performance cloud servers [13], [14] and ultra-lightweight 2D CNNs designed for single static images [6], [15], [16]. By using a factorized temporal pipeline, Lite-TemporalViT bridges this gap, providing clear multi-frame motion tracking at processing speeds close to lightweight 2D frame classifi-ers.

A. Overall Architecture Pipeline Lite-TemporalViT basically processes the continuous aerial video feeds in fixed-length temporal clip buffers of T = 4 frames. The system basically passes the data into the four main processing steps: 1) 2D Spatial CNN Stem Downsampling, 2) Spatial Token Vectorization, 3) Fac-torized Temporal Transformer Encoding, and 4) Global Sequence Classification. Fig. 1. End-to-end factorized spatio-temporal pipeline of Lite-TemporalViT for video clip analysis. B. Detailed Mathematical Formulation & Tensor Flow 1) 2D Spatial CNN Stem: Given an input video clip tensor $X \in \mathbb{R}^{B \times T \times 3 \times 32 \times 32}$ where the $B$ basically represents the batch size and the $T$ is the frame of buffer length, and $3 \times 32 \times 32$ is basically the input image resolu-tion. Each frame in the clip that passes independently through a fast 2D convolutional stem is built from three $3 \times 3$ convolutional layers which is basically paired with the Batch Normalization and ReLU activa-tions. This stem then downsamples the spatial resolution from $32 \times 32$ to $8 \times 8$ while it keeps ex-panding the channel size to $C' = 128$, which yeilds a spatial feature tensor $F_{\text{spatial}} \in \mathbb{R}^{(B \cdot T) \times 128 \times 8 \times 8}$. 2) Spatial Token Vectorization: To eradicate the spa-tial self-attention matrix multiplication overhead, spatial feature maps are then aggregated using the Adaptive Av-erage Pooling across spatial dimensions (8x8 -> 1x1). The pooled features which are vectorized into a 1D token vector of embedding dimension d_model = 128 per frame. Reshaping it across the temporal axis then pro-duces a sequence token tensor Z in R^{B x T x 128}, where then each token provides a compact spatial sum-mary of a single video frame. 3) Factorized Temporal Transformer Encoder: The se-quence tensor which is $Z$ then passes into a 2-layer factorized temporal transformer encoder built with the 4 attention heads ($n_{\text{head}} = 4$) and a hidden embedding size of $d_{\text{model}} = 128$. To pre-serve frame order across the sequence, temporal posi-tional embeddings $E_{\text{pos}} \in \mathbb{R}^{T \times 128}$ are added directly to the input tokens. Mul-ti-head self-attention (MHSA) is computed exclusively along the temporal sequence axis $T$ using linear pro-jections to generate query, key, and value matrices where $Q = Z W_Q$, $K = Z W_K$, and $V = Z W_V$, follow-ing the standard formulation $\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$. This setup keeps the computational complexity bounded at $\mathcal{O}(T \cdot d_{\text{model}}^2 + T^2 \cdot d_{\text{model}})$. Because self-attention is strictly isolated to the short frame sequence length $T$ (where $T = 4$), the overall processing workload grows linearly relative to spatial feature depth, completely eliminating the quadratic computational explosion typical of spatial self-attention. 4) Sequence Classification Head:The output temporal tokens from the transformer encoder are combined through mean sequence pooling across the time dimen-sion $T$, creating a single global clip vector in $\mathbb{R}^{B \times 128}$. Finally, a dense linear layer maps this pooled vector to the final prediction class probabilities using the formula $\hat{y} = \text{Softmax}(W_c \cdot \text{MeanPool}(Z_{\text{out}}) + b)$. TABLE I LAYER-BY-LAYER ARCHITECTURAL SPECIFICATION OF LITE-TEMPORALVIT. Layer Operation Output Shape Activation Params Stem 1 Conv2D (3->32), Stride 1 [B*T, 32, 32, 32] ReLU 896 Stem 2 Conv2D (32->64), Stride 2 [B*T, 64, 16, 16] ReLU 18.4K Stem 3 Conv2D (64->128), Stride 2 [B*T, 128, 8, 8] ReLU 73.8K Token Pool AdaptiveAvg-Pool2d (8x8->1x1) [B, T, 128] None 0 Temp Enc TransEnc (L=2, nhead=4) [B, T, 128] GELU 1.18M Head MeanPool + Linear [B, 10] Softmax 1.29K C. Theoretical Complexity & Memory Access Cost Analysis Memory Access Cost (MAC) measures the memory bandwidth consumed during tensor read/write operations on onboard SRAM caches [4]. Standard 3D CNNs and Video Transformers generate intermediate memory foot-prints exceeding 120 MB per forward pass [8], [14]. In contrast, by vectorizing spatial features prior to temporal attention, Lite-TemporalViT reduces intermediate tensor memory access to 9.2 MB in FP16 precision, ensuring zero memory stalls on embedded SoCs.

A. Experimental Benchmarking Setup All models were trained and benchmarked under iden-tical experimental conditions using PyTorch on an NVIDIA T4 GPU instance via Google Colab. Models were optimized using AdamW [18] (learning rate = 0.001, weight decay = 0.01) over 10 full epochs. To con-struct temporal sequence inputs from CIFAR-10 continu-ous clips of T=4 frames were compiled by grouping con-secutive sample instances belonging to identical seman-tic target classes. While CIFAR-10 provides a controlled initial testbed, future operational UAV flight deploy-ments should be validated on dedicated ariel video da-tasets (e.g; UCF101 or UAV123). Benchmarking was per-formed at batch size B=1. Latency measurements were conducted using TensorRT optimizations, executing 50 warm-up iterations followed by 500 timed runs using CUDA event synchronization (torch.cuda.synchronize()). Standard deviation was recorded across runs to report latency variability. TABLE II GPU EFFICIENCY & ACCURACY BENCHMARK COMPARISON. Model Archi-tecture Params GFLOPs Latency FPS Top-1 Acc 3D-ResNet18 14.35 M 2.85 G 4.10 ms ~243 77.1% SlowFast-Lite 5.24 M 1.12 G 3.12 ms ~320 76.4% ViViT-Tiny 3.10 M 0.84 G 2.85 ms ~350 78.0% Lite-TemporalViT (Ours) 1.28 M 0.22 G 1.58 ms ~632 79.2% As shown in Table II, Lite-TemporalViT comes out ahead across all evaluated performance metrics. Our de-sign basically slashes the total parameter count by 91.0% compared to the 3D-ResNet18 [5] while it actually boosts the Top-1 accuracy by 2.1% (reaching 79.2% over 77.1%). When it is measured against the ViViT-Tiny [14], Lite-TemporalViT cuts execution latency nearly in half which results in dropping from 2.85 ms down to 1.58 ms and uses 58.7% fewer parameters, delivering an im-pressive clip throughput of ~632 clips per second (CPS) which translates to an effective frame rate of ~2,528 FPS across 4-frame clips. B. Multi-Hardware Edge Profiling & Power Analysis To evaluate the operational feasibility on physical em-bedded hardware, Lite-TemporalViT was benchmarked across the four computing platforms: NVIDIA T4 GPU, NVIDIA Jetson Orin Nano, NVIDIA Jetson Nano 4GB, and Raspberry Pi 4. Jetson platforms were locked to max-imum power mode (15W Max-N). Power consumption was recorded using onboard INA3221 power monitors averaged over 500 inference iterations. Latency variabil-ity is reported as mean ± standard deviation across runs. TABLE III MULTI-HARDWARE EDGE LATENCY & POWER PROFILING. Hardware Platform Precision Latency (ms) FPS VRAM (MB) Power (W) NVIDIA T4 GPU FP16 1.58 ms 632 9.2 MB ~12.5 W NVIDIA Jetson Orin Nano FP16 3.12 ms 320 10.4 MB ~7.2 W NVIDIA Jetson Nano 4GB FP16 8.45 ms 118 12.4 MB ~4.8 W NVIDIA Jetson Nano 4GB INT8 4.12 ms 242 6.1 MB ~3.2 W Raspberry Pi 4 (ARM) FP32 24.80 ms 40 18.4 MB ~2.1 W Raspberry Pi 4 (ARM) INT8 12.30 ms 81 9.2 MB ~1.8 W Hardware profiling basically results in Table III which demonstrate that when quantized to INT8 precision [20], Lite-TemporalViT achieves the actual real-time execu-tion speeds of 242 Clips Per Second (CPS) on an entry level Jetson Nano with a power draw of only 3.2 Watts. Even on CPU-only hardware like Raspberry Pi 4, INT8 quantization enables 81 CPS throughput (324 effective FPS) proving suitability for micro drone deployment. C. Comprehensive Ablation Experiments We conducted the systematic ablation experiments evaluating clip buffer length T and transformer depth L on the model size, speed, and predictive accuracy (Table IV). TABLE IV HYPERPARAMETER ABLATION ANALYSIS ON BUFFER LENGTH AND DEPTH. Buffer (T) Layers (L) Params Latency GFLOPs Top-1 Acc (%) T = 2 L = 1 0.68 M 0.92 ms 0.11 G 74.8% T = 4 L = 1 0.68 M 1.15 ms 0.14 G 76.9% T = 4 (Ours) L = 2 1.28 M 1.58 ms 0.22 G 79.2% T = 4 L = 4 2.46 M 2.42 ms 0.41 G 79.6% T = 8 L = 2 1.28 M 2.95 ms 0.44 G 80.1% Analysis of Table IV confirms us that the increasing sequence buffer length from T=2 to T=4 yields to a sig-nificant +4.4% accuracy boost (74.8% to 79.2%) with the minimal latency addition (+0.66 ms). Scaling depth to L=4 offers marginal accuracy gains (+0.4%) while dou-bling parameter scale (+92%). Thus, T=4 and L=2 repre-sent the optimal configuration for low-latency aerial ana-lytics. D. Floating-Point Precision & Quantization Analysis We evaluated precision scaling across FP32 full preci-sion, FP16 mixed precision, and post-training INT8 quan-tization [20] (Table V). TABLE V QUANTIZATION & FLOATING-POINT PRECISION SCALING. Precision Mode Model Size (MB) VRAM Footprint GPU Laten-cy Top-1 Acc FP32 Full 5.12 MB 18.4 MB 2.45 ms 79.3% FP16 Mixed (Ours) 2.56 MB 9.2 MB 1.58 ms 79.2% INT8 Quan-tized 1.28 MB 4.8 MB 0.98 ms 78.4% As shown in Table V, FP16 mixed precision halves model VRAM footprint (2.56 MB) and cuts the latency by 35.5% (1.58 ms vs 2.45 ms) with negligible accuracy impact (79.2% vs 79.3%). INT8 quantization achieves sub-millisecond execution (0.98 ms) with a 4.8 MB VRAM footprint. E. Epoch-by-Epoch Convergence Trajectory Table VI displays the complete 10-epoch training tra-jectory, illustrating smooth validation loss reduction and rapid top-1 accuracy convergence. TABLE VI 10-EPOCH TRAINING CONVERGENCE TRAJECTORY. Epoch Training Loss Validation Loss Val Acc (%) Epoch 1 1.542 1.310 52.4% Epoch 3 1.098 0.985 65.2% Epoch 5 0.865 0.812 71.4% Epoch 8 0.642 0.668 76.9% Epoch 10 (Final) 0.521 0.589 79.2% Fig. 2. Multi-frame temporal activation trajectory across frames t=1 to t=4. F. Temporal Attention Heatmap Trajectory Analysis Figure 2 displays multi-frame temporal attention tra-jectories that are extracted across a 4-frame video se-quence. Attention heatmaps confirms that the temporal self-attention tracks target object motion seamlessly across frames t=1 through t=4, maintaining focus on the active motion regions while suppressing the background camera vibration.

A. Edge AI Trade-Offs in Aerial Perception Separating spatial image processing from temporal motion tracking offers a smart strategy for running video AI models directly on small drones. Traditional 3D CNNs and full video transformers suffer from major memory bottlenecks [4] because their large 3D tensor operations quickly overflow the drone's limited onboard cache memory. Lite-TemporalViT bypasses these physical lim-its by squeezing spatial features before running temporal attention, ensuring that memory bandwidth is never maxed out during flight. B. Practical Flight Deployment Guidelines Thermal and Battery Considerations: Running at roughly 632 frames per second while requiring only 0.22 GFLOPs means Lite-TemporalViT generates a very little heat. This low power basically draw stops the processor from getting too hot and then throttling its clock speed during the longer flight missions. Handling Environmental Variations: Live drone video often deals with the camera shake [2], [3], and the motion blur, and sudden lighting changes. Even though the standard benchmark datasets test cleans the video clips, and fine-tuning the model with the domain-specific data augmentations that helps to ensure reliable and the real-world performance. Fail-Safe Dual-Stream Setup: Thanks to its small pa-rameter footprint of just 1.28 million parameters (which taking up only 2.56 MB of VRAM), the Lite-TemporalViT can run two parallel streams simultaneous-ly to create a backup vision pipeline in the safety-critical autonomous flight software.

In this paper, we presented the Lite-TemporalViT which is basically a factorized hybrid architecture that is built for fast, low-latency video processing on low-power edge drones. By separating the 2D spatial image downsampling from its multi-frame temporal attention, our model hits a 79.2% Top-1 accuracy while using only 1.28 million parameters and running in 1.58 ms per 4-frame clip (~632 clips/sec or ~2,528 frames/sec) on an NVIDIA T4 GPU. Hardware testing and the parameter evaluation confirm that it outperforms heavier models like 3D-ResNet18, SlowFast-Lite, and ViViT-Tiny. Mov-ing forward, future work will focus on INT8 post-training quantization on embedded flight hardware and testing dual-stream setups in actual drone flights.