Performance Evaluation of Scalable Computing Systems for Large Language Model Inference

Large language models (LLMs) impose substantial memory and serving requirements, making locally controlled inference increasingly relevant for research laboratories, universities, and enterprises that require data governance, offline operation, reproducibility, customization, and predictable infrastructure cost. In such environments, performance depends not only on compute capability but also on the interaction among model weights, runtime buffers, KV caches, concurrent requests, and system memory. NVIDIA DGX Spark provides a compact Grace Blackwell platform with 128 GB of unified memory, 273 GB/s memory bandwidth, and ConnectX-7 networking [1]. Because CPU and GPU allocations share the same memory pool, its scaling behavior differs from that of conventional discrete-GPU servers. This study evaluates OpenAI gpt-oss-120b, a Mixture-of-Experts model with 116.83 billion total parameters, 5.13 billion active parameters per token, and an approximately 60.8 GiB checkpoint [2], [3]. The central question is whether adding a second DGX Spark provides a meaningful inference advantage when the same model can already run on a single node. Dual-node execution increases compute and memory capacity but also introduces pipeline, communication, and synchronization overheads, so near-linear scaling cannot be assumed. This paper makes four main contributions. First, it establishes a reproducible Single- versus Dual-node DGX Spark testbed using identical models, serving configurations, and workloads. Second, it implements native multi-node vLLM inference with TP=1 and PP=2 over ConnectX-7. Third, it evaluates four workload types across five concurrency levels using TTFT, generation throughput, end-to-end latency, tail latency, aggregate throughput, GPU utilization, and unified-memory utilization. Finally, it identifies the workload conditions in which Dual Mode is beneficial and translates the measured results into practical deployment guidance for closed local-LLM services.

DGX Spark integrates Grace Blackwell compute with 128 GB LPDDR5x unified system memory. The memory has a 256-bit interface and 273 GB/s bandwidth, while the system includes a ConnectX-7 Smart NIC [1]. In this study, the two Spark nodes remain separate memory domains; model parallelism partitions execution across nodes rather than presenting a transparent 256-GB shared-memory machine.

OpenAI gpt-oss-120b is an open-weight Mixture-of-Experts (MoE) Transformer [2], [3]. Its open-weight deployment model enables the model to be operated on user-controlled computing infrastructure, making it suitable for institution-controlled and closed local-LLM environments [4]. MoE sparsity reduces the number of parameters activated per token relative to the total parameter count, but the full checkpoint still determines model storage and loading requirements.

vLLM is designed for high-throughput LLM inference and uses PagedAttention to manage KV-cache memory efficiently [5]. Model parallelism, including tensor and pipeline parallelism, has been widely used to partition large Transformer models across multiple accelerator resources [6]. Current vLLM supports tensor and pipeline parallelism as well as both Ray and native multiprocessing distributed backends. For multi-node multiprocessing, nodes are assigned explicit ranks and a master address; the pipeline-parallel size can be set to the number of nodes [7]. This matches the physical topology of the testbed, where each DGX Spark contributes one GPU execution domain. NCCL provides topology-aware GPU communication primitives, while PyTorch distributed uses NCCL for CUDA collectives and Gloo for CPU-side coordination [8], [9].

The baseline runs one vLLM API server and one gpt-oss-120b model instance on Spark 1. The distributed executor backend is native multiprocessing, but the parallelism configuration is TP=1 and PP=1 with nnodes=1. No inter-node synchronization is required. The single-node configuration therefore establishes the reference for both interactive and capacity-oriented metrics.

The final dual-node system uses vLLM native multiprocessing rather than Ray/KubeRay. Spark 1 acts as rank 0, hosts the OpenAI-compatible API server, and executes pipeline stage 0. Spark 2 acts as rank 1 in headless mode and executes pipeline stage 1. The configuration is TP=1, PP=2, and nnodes=2. The high-speed ConnectX-7 interfaces use link-local IPv4 addresses 169.254.199.119 (Spark 1) and 169.254.80.141 (Spark 2). VLLM_HOST_IP, NCCL_SOCKET_IFNAME, and GLOO_SOCKET_IFNAME are explicitly set so that model-parallel and coordination traffic use the ConnectX-7 path rather than the management network. The use of pipeline parallelism introduces stage-to-stage communication and pipeline scheduling overhead. For a request $r$, the observed response time can be decomposed conceptually into prefill, decode, inter-node communication, synchronization, and queueing components as follows.

\[T_{\mathrm{E2E}}(r)=T_{\mathrm{prefill}}(r)+T_{\mathrm{decode}}(r)+T_{\mathrm{comm}}(r)+T_{\mathrm{sync}}(r)+T_{\mathrm{queue}}(r)\tag{1}\]

Here, Tprefill denotes prompt-processing time, Tdecode denotes output-token generation time, Tcomm captures inter-node transfer cost, Tsync represents distributed synchronization overhead, and Tqueue is request waiting time. This decomposition explains why a second node can reduce compute or memory pressure while still increasing per-request overhead through communication and synchronization.

Both systems use the NVIDIA vLLM container nvcr.io/nvidia/vllm:26.08-py3 with vLLM 0.27.1-series software. The model is openai/gpt-oss-120b. The maximum model length is fixed at 6,144 tokens, max_num_seqs at 16, and gpu_memory_utilization at 0.65 for both Single and Dual. The common memory target matters because an earlier 0.70 configuration exceeded the worker node's free unified-memory headroom; therefore, all final comparative results in this paper use the re-measured 0.65 condition. Eager execution is enabled, and both configurations expose the same OpenAI-compatible endpoint to the benchmark driver.

Four workload classes are generated by combining short and long inputs with short and long output targets. The benchmark uses concurrency levels C={1,2,4,8,16}, and each workload/concurrency condition is repeated three times. Thus, each full configuration contains 4×5×3=60 summarized experimental runs, in addition to per-request records. Concurrency 1–4 uses 20 requests per repeat, concurrency 8 uses 32 requests, and concurrency 16 uses 64 requests. Before measurement, the serving process is warmed up. The benchmark client sends concurrent streaming requests and records request start, first-token arrival, final-token arrival, completion-token count, TTFT, generation duration, end-to-end latency, and per-request generation rate. Raw request data and one summary row per workload/concurrency/repeat combination are stored in CSV files. We sample GPU utilization and unified-memory usage independently at approximately one-second intervals during the run. For Dual, we collect telemetry on both nodes and combine it into a per-node sample distribution. To provide a compact system-level comparison, the representative Table 3 condition is W4 at concurrency 16 with three repeats. This is the most demanding workload in the matrix and stresses prompt processing, decoding, KV-cache occupancy, and concurrent scheduling simultaneously.

Table I reports the final representative comparison. Each latency and throughput value is the mean of three W4/C=16 repeats. GPU and memory values are derived from the telemetry samples collected during the corresponding dedicated runs. Dual GPU utilization is the sample-weighted mean across both nodes; therefore it should be interpreted as average utilization per active node, not as total system compute consumption. The representative run shows a mixed but practically useful scaling profile. Mean TTFT decreases from 0.441 s to 0.394 s, a 10.52% improvement. Mean generation throughput increases by 13.78%, and aggregate throughput rises from 185.61 to 211.45 tokens/s, corresponding to a 13.92% gain, a 1.139× throughput speedup, and 56.96% two-node parallel efficiency. The efficiency is far below 100%, confirming that the second node does not translate directly into twice the service rate. Tail behavior improves more than the mean. P99 E2E latency falls from 33.26 s to 28.64 s (13.91% improvement), while P95 improves by 2.33%. However, mean E2E latency increases from 15.37 s to 16.40 s, a 6.69% degradation. This combination indicates that pipeline parallelism improves service capacity and the worst tail cases but introduces enough per-request pipeline/coordination cost to increase the arithmetic mean under this condition. Average GPU utilization remains high in both cases (94.49% Single versus 93.48% per-node Dual). Peak GPU utilization reaches 96% in both cases. Mean per-node unified-memory utilization decreases from 75.78% to 73.86%. This does not imply that Dual consumes less total memory; rather, work is distributed over two independent 128-GB unified-memory domains, increasing aggregate physical capacity while reducing average pressure per node.

The full workload matrix demonstrates that scaling behavior depends strongly on output length and concurrency. Figure 2 plots aggregate throughput against concurrency for W1–W4. The short-output W1 case derives little benefit from the second node and becomes slower at high concurrency. By contrast, W2 and W3 show clear gains at concurrency 16, while W4 shows a consistent throughput advantage across the tested concurrency range, although the size of the gain varies.

At C=16, W1 loses 10.85% aggregate throughput and exhibits worse TTFT, generation rate, and mean E2E latency. This indicates that the pipeline and inter-node overhead are not amortized when both the input and output are short. W2 gains 17.80% aggregate throughput, improves TTFT by 12.40%, increases generation throughput by 17.19%, and reduces mean E2E latency by 14.37%. W3 shows the strongest high-concurrency benefit: aggregate throughput increases by 28.87%, generation throughput by 22.10%, TTFT improves by 16.38%, and mean E2E latency improves by 32.39%. W4 gains 12.24% aggregate throughput and 8.97% generation throughput, but its mean E2E latency is 17.59% worse in the full-matrix C=16 runs. These results show that long decode phases can provide enough work to amortize pipeline overhead, but long prompts and long outputs together can still expose stage imbalance and queuing effects.

TTFT reflects both prompt-prefill cost and queueing before the first decode token. In W4, Dual TTFT is close to Single at low concurrency and becomes lower from concurrency 4 onward. However, E2E latency tells a different story: Dual is substantially slower at low W4 concurrency and remains above Single at C=16 in the full matrix. This is consistent with a pipeline-parallel system in which a single request cannot fully exploit both stages simultaneously, while concurrency allows the pipeline to fill and raises total service throughput. Therefore, TTFT and aggregate throughput should not be used as substitutes for E2E latency; each captures a different aspect of user-perceived performance.

The data do not support a single universal crossover concurrency. W1 briefly favors Dual at C=4 but then falls below Single at C=8 and C=16. W2 is approximately equal at C=1, slower at C=2, faster at C=4, slower at C=8, and clearly faster at C=16. W3 follows a similar non-monotonic pattern, with strong gains at C=4 and C=16 but a loss at C=8. W4 is the most consistently throughput-positive case. The practical implication is that routing policy should be workload-aware rather than based on concurrency alone.

This paper experimentally characterizes single- and dual-node DGX Spark inference for gpt-oss-120b under controlled, identical serving parameters. The final dual-node implementation uses vLLM native multiprocessing with TP=1 and PP=2 over ConnectX-7. The results reject the assumption that a second node produces uniform or near-linear speedup. In the representative W4/C=16 condition, Dual improves mean TTFT by 10.52%, mean generation throughput by 13.78%, aggregate throughput by 13.92%, and P99 E2E latency by 13.91%, but mean E2E latency worsens by 6.69%. Across the full C=16 matrix, throughput ranges from a 10.85% loss on W1 to a 28.87% gain on W3. The key finding is therefore workload selectivity: dual-node pipeline parallelism is most valuable when concurrency and generation length are sufficient to amortize communication and pipeline overhead. From a service-deployment perspective, Single mode is better aligned with latency-oriented interactive local assistants, whereas Dual mode is better aligned with long-generation, document-centric, and high-concurrency services. A closed local-LLM platform can therefore benefit from workload-aware routing that selects the execution topology according to prompt length, expected output length, queue state, and service-level objectives. These findings provide both a quantitative scaling characterization and practical guidance for deploying gpt-oss-120b on compact institution-controlled infrastructure.

Acknowledgement. This research was supported by the Gyeongbuk ANCHOR system through the Gyeongbuk ANCHOR CENTER, funded by the Ministry of Education(MoE) and Gyeonsangbuk-do, Republic of Korea (2026-ANCHOR-15-107).