From Benchmark Saturation to Model Discrimination: Evaluating Ten Open-Source Embedding Models for Thai Dense Retrieval

Dense text embeddings are a core component of Retrieval-Augmented Generation (RAG), where they connect user queries with external knowledge supplied to a generative model [1]. Their effectiveness depends not only on the pretrained encoder but also on the evaluation setting used to compare candidate systems. This issue is particularly relevant for Thai, where word boundaries are not consistently marked by whitespace and retrieval pipelines may require language-aware preprocessing before model-specific tokenization. Although multilingual benchmarks such as MTEB [2] provide valuable evidence across languages and tasks, a Thai-specific retrieval benchmark can still produce misleading conclusions if its candidate pool is too shallow relative to the evaluation depth. When too few competing documents are available, retrieval metrics can approach their upper bounds even when meaningful model differences exist. Benchmark saturation therefore becomes a methodological problem rather than merely a property of the evaluated models. This study was motivated by such saturation in an initial Thai dense-retrieval configuration. The benchmark was subsequently strengthened to 60 queries and 120 candidate documents across eight broad domains, with multiple topically related documents providing more realistic competition. A complementary Semantic Textual Similarity (STS) track contains 150 predefined Thai sentence pairs. The revised setting is designed to reveal whether increasing retrieval difficulty improves model discrimination while exposing any remaining model-specific ceiling effects. We evaluate ten open-source multilingual, regional, Thai-native, and lightweight embedding systems under a common controlled protocol. The comparison considers three dimensions: dense-retrieval effectiveness, STS behavior, and CPU inference efficiency. The study is intended as a controlled investigation of benchmark design and practical model-selection trade-offs rather than a definitive leaderboard for Thai retrieval. The research gap addressed here is the limited attention given to the interaction between benchmark depth and observable model discrimination in controlled Thai dense retrieval. Existing Thai and Southeast Asian resources provide broader task coverage, whereas the present work focuses specifically on how benchmark structure can conceal or expose differences among embedding systems under a common inference setting. The study makes four contributions: (1) it diagnoses structural saturation in an initially shallow Thai retrieval setting and strengthens the candidate space; (2) it compares ten open-source embedding systems using retrieval and STS measures; (3) it provides a reusable Thai-oriented ingestion and evaluation harness while separating optional preprocessing from the fixed controlled benchmark; and (4) it reports tokenizer-based CPU throughput to expose accuracy-efficiency trade-offs relevant to Thai RAG deployment.

Dense retrieval commonly represents queries and passages in a shared vector space, as exemplified by Dense Passage Retrieval (DPR) [3]. BERT [4] established a general pretrained Transformer foundation, and Sentence-BERT [5] adapted this paradigm for efficient sentence-level semantic representation. More recent multilingual embedding systems include BGE-M3 [6] and the E5 family [7], while regional and language-focused alternatives include SEA-LION [8], ModernBERT-based regional encoders [9], and the Thai-native WangchanBERTa model [10]. Their different objectives, tokenizers, corpora, and sentence-embedding adaptations motivate direct comparison under a common Thai retrieval protocol. Thai and Southeast Asian embedding evaluation has expanded in recent years. The Thai Sentence Embedding Benchmark [11] covers multiple sentence-embedding tasks, the Thai Sentence Vector Benchmark [12] evaluates Thai sentence representations in zero-shot and transfer settings, and SEA-BED [13] contributes regional datasets for Southeast Asian languages. NitiBench [14] further demonstrates applied embedding evaluation in Thai legal question answering. These resources provide important broader reference points for Thai representation quality. The present study is complementary to these larger benchmarks. Its specific focus is benchmark saturation: whether a controlled retrieval collection provides sufficient candidate competition to distinguish models meaningfully. By combining a strengthened retrieval testbed with STS and tokenizer-based CPU throughput under a common inference protocol, the study examines both model discrimination and practical deployment trade-offs without claiming to replace existing large-scale Thai or multilingual benchmarks.

This section describes the architecture of the evaluation framework and the experimental methodology used to compare the ten embedding systems. The framework combines Thai-oriented ingestion capabilities, a strengthened dense-retrieval benchmark, semantic textual similarity evaluation, and standardized inference profiling. To ensure comparability, all systems are evaluated on the same fixed retrieval and STS collections under a common raw-text inference protocol. The following subsections summarize the candidate systems, preprocessing capabilities, benchmark construction, evaluation metrics, and inference settings used in the study.

We select ten representative systems spanning high-capacity multilingual encoders, regional models, a Thai-native foundation-model baseline, general multilingual sentence encoders, and lightweight distilled architectures. Their output dimensionalities range from 384 to 1024, with parameter counts from approximately 22M to 568M. Unlike space-delimited Latin scripts, Thai text processing introduces additional tokenization and document-segmentation considerations because lexical boundaries are not consistently represented by whitespace. Raw Thai text obtained from heterogeneous sources, including web pages, parsed PDF documents, CSV files, JSON records, and governmental or institutional repositories, may also contain invisible formatting artifacts and irregular whitespace. The benchmarking harness therefore includes a reusable Thai-oriented ingestion layer designed to prepare external document collections before embedding generation. These capabilities are summarized in the following subsection.

The benchmarking harness includes reusable Thai-oriented ingestion capabilities for external or long-form corpora. Unicode sanitization removes configured zero-width and formatting artifacts and normalizes whitespace. Thai lexical boundaries can be generated with PyThaiNLP newmm maximal matching, with a regex-based fallback when the dictionary tokenizer is unavailable. An optional sliding-window chunker creates overlapping retrieval units while preserving word boundaries, and a corpus dispatcher supports plain-text, CSV, and JSON inputs. These capabilities are distinct from the controlled benchmark reported here. The fixed 120 candidate documents were supplied directly to each evaluated system without additional sliding-window subdivision. The same 60 retrieval queries and 150 STS pairs were held fixed across systems, separating reusable ingestion functionality from the benchmark instances used to generate the reported results.

The retrieval collection is distributed across eight broad domains: environment and climate; law, regulation, and governance; artificial intelligence, computer science, and NLP; medicine, health, and nutrition; finance, economy, and business; education and academic guidelines; culture, tourism, history, and Thai society; and science, transport, and infrastructure. Each query has one or more predefined relevant document identifiers, while multiple documents within a domain provide semantically plausible competing candidates. Retrieval effectiveness is evaluated using NDCG@10, MRR@10, MAP, Recall@10, and Precision@10. These metrics capture ranked-list quality, the position of the first relevant item, precision over relevant items, and top-ten retrieval coverage.

The retrieval setting was strengthened from the initial shallow testbed by expanding the evaluation space to 60 queries and 120 candidate documents. The expanded collection contains multiple documents within the same broad domains and includes candidates that share topical terminology or partially related semantic content with relevant documents. This structure increases the opportunity for competing candidates to receive relatively high embedding similarity, thereby making the retrieval task more discriminative than the initial shallow configuration. The expanded candidate collection includes multiple documents within the same broad domains, so retrieved items may share terminology or related concepts without directly satisfying the query. This creates more realistic competition for dense retrieval and gives ranking errors greater opportunity to affect the evaluation metrics.

The benchmarking harness (v1.0.0) was executed in an isolated CPU-only environment on a 10-core host with 16.0 GB RAM. All ten candidate models were evaluated sequentially against the same controlled benchmark instances: 120 candidate documents and 60 retrieval queries for dense retrieval, and 150 predefined sentence pairs for Semantic Textual Similarity (STS). The benchmark collections and relevance definitions were held fixed across models, and the same cosine-similarity calculation, retrieval metrics, and STS correlation procedures were applied throughout the comparative evaluation. Only outputs generated from successfully loaded pretrained model weights were retained in the final empirical results. The final evaluation therefore excludes fallback or simulated embedding outputs. For the controlled retrieval experiment, the fixed 120-document collection was supplied directly to each model without additional sliding-window subdivision. The reusable ingestion and chunking capabilities described in Section III.B remain available for external and longer-form corpora but are distinct from the fixed benchmark instances used to generate the results reported here.

The strengthened retrieval evaluation produces a wide performance distribution across the ten candidate models. NDCG@10 ranges from 0.4783 to 1.0000, MRR@10 from 0.4311 to 1.0000, MAP from 0.4291 to 1.0000, and Recall@10 from 0.6750 to 1.0000. Precision@10 ranges from 0.0700 to 0.1050. These ranges demonstrate that the strengthened benchmark no longer exhibits the across-model structural saturation observed in the initial shallow configuration. GTE Multilingual Base achieved the highest observed retrieval performance, reaching NDCG@10 = 1.0000, MRR@10 = 1.0000, MAP = 1.0000, and Recall@10 = 1.0000. Its Precision@10 was 0.1050. Under the relevance structure of the present controlled benchmark, these results indicate that all relevant documents were retrieved within the evaluation depth and placed in ideal ranking positions for the evaluated queries. The result should therefore be interpreted as benchmark-specific ceiling performance rather than evidence of universally perfect Thai retrieval. BGE-M3 and Multilingual-E5 Large formed the next closely matched high-capacity group. BGE-M3 achieved NDCG@10 = 0.8173, MRR@10 = 0.8029, MAP = 0.7910, and Recall@10 = 0.9000. Multilingual-E5 Large produced nearly identical values, with NDCG@10 = 0.8164, MRR@10 = 0.8019, MAP = 0.7901, and Recall@10 = 0.9000. Multilingual-E5 Base, Paraphrase Multilingual MPNet, LaBSE, and SEA-LION ModernBERT also remained within a relatively narrow retrieval region, with NDCG@10 values between 0.8011 and 0.8081. WangchanBERTa produced a different pattern from the retrieval-tuned multilingual encoders. Its final live-weight evaluation resulted in NDCG@10 = 0.5943, MRR@10 = 0.5461, MAP = 0.5532, Recall@10 = 0.7500, and Precision@10 = 0.0767. These values place it below the major multilingual and regional embedding models in the present retrieval task and emphasize the importance of evaluating the actual pretrained model rather than inferring retrieval suitability from Thai-specific pretraining alone. The two 384-dimensional distilled models also showed lower retrieval effectiveness than most of the larger multilingual embedding models. Paraphrase Multilingual MiniLM-L12 achieved NDCG@10 = 0.6379, MRR@10 = 0.5974, MAP = 0.5979, and Recall@10 = 0.7917. The All-MiniLM-L6-v2 control produced the lowest retrieval scores in the evaluated pool, with NDCG@10 = 0.4783, MRR@10 = 0.4311, MAP = 0.4291, and Recall@10 = 0.6750.

The STS evaluation provides a complementary view of the embedding spaces. Spearman's ρ ranges from 0.2790 to 0.9287 and Pearson's r from 0.2768 to 0.9229. GTE Multilingual Base again occupies a distinct upper region, reaching ρ=0.9287 and r=0.9229. BGE-M3 and Multilingual-E5 Large form a second closely matched group at ρ=0.5158 and 0.5154, while E5-Base, SEA-LION, MPNet, and LaBSE cluster around ρ≈0.50. WangchanBERTa records the lowest STS correlation, and the MiniLM models remain below the stronger multilingual encoders.

Tokenizer-based CPU throughput ranges from 217.16 to 2,431.48 tokens/s. BGE-M3 and E5-Large are the slowest measured systems in the CPU environment, at 217.16 and 245.57 tokens/s. GTE reaches 724.12 tokens/s while achieving the strongest observed retrieval and STS results. E5-Base, MPNet, and LaBSE operate near 900 tokens/s, whereas the 384-dimensional MiniLM models exceed 2,300 tokens/s but provide lower retrieval effectiveness. Tokenizer-based throughput is more defensible than the preliminary character-count proxy, but it remains model dependent because different tokenizers can segment the same Thai text into different numbers of subword units. The reported values should therefore be interpreted as observed model-specific inference behavior under a common textual workload rather than universal hardware-normalized speed constants.

This section interprets the experimental findings beyond the reported metric values, with particular emphasis on benchmark behavior and cross-model trade-offs. The discussion examines how the strengthened retrieval setting changes model discrimination, how architectural and training differences relate to observed performance, and why small numerical gaps should be interpreted cautiously. Practical implications and key limitations of the controlled benchmark are also considered to clarify the scope of the conclusions.

The principal methodological finding is the distinction between structural benchmark saturation and model-specific ceiling performance. The initial shallow setting compressed differences because the candidate pool provided too few opportunities for ranking errors to become visible. After expansion, NDCG@10 spans 0.4783–1.0000 and Recall@10 spans 0.6750–1.0000, demonstrating substantially greater model discrimination across the evaluated pool. GTE still reaches the upper bound, showing that saturation is not binary. A benchmark can separate most systems while remaining too easy for a particularly well-matched model. GTE's perfect retrieval scores therefore indicate ceiling performance on this controlled collection rather than universally perfect Thai retrieval. Larger candidate sets, harder within-domain competitors, and more varied query forms are needed to determine whether this ceiling persists. The broader implication is methodological: benchmark discrimination depends on candidate diversity, query formulation, relevance structure, semantic overlap, annotation quality, and retrieval depth. The present testbed should therefore be viewed as a controlled methodological investigation rather than a final Thai retrieval leaderboard.

The final live-model results also show that simple architectural descriptors are poor substitutes for direct evaluation. All 768-dimensional systems do not behave alike: GTE reaches NDCG@10=1.0000; E5-Base, MPNet, LaBSE, and SEA-LION remain near 0.80; and WangchanBERTa reaches 0.5943. These differences reflect complete pretrained systems with different objectives, tokenizers, corpora, and sentence-embedding adaptation rather than vector dimensionality alone. WangchanBERTa is informative as a Thai-native foundation-model baseline. It was developed as a Thai pretrained language model rather than a retrieval-specialized contrastive sentence encoder. Its lower retrieval and STS results should therefore not be interpreted as evidence against Thai-specific language modeling; instead, they illustrate the distinction between language-specific pretraining and off-the-shelf sentence-retrieval specialization. The MiniLM models illustrate a different operating point. Their 384-dimensional outputs reduce vector size and their CPU throughput exceeds 2,300 tokens/s, but retrieval effectiveness is lower. BGE-M3 and E5-Large offer strong retrieval at substantially lower CPU throughput, while E5-Base, MPNet, and LaBSE provide intermediate operating points near 900 tokens/s with NDCG@10 around 0.80. Model selection should therefore consider retrieval quality, semantic behavior, vector dimensionality, computational resources, tokenizer behavior, memory, and compatibility with the target corpus.

Fine-grained ordering among closely matched systems is not established. BGE-M3 and E5-Large differ by only 0.0009 in NDCG@10, and several 768-dimensional encoders differ by only a few thousandths. Because the study does not report a validated paired significance test, these small numerical differences are interpreted descriptively rather than as reliable superiority.

Several limitations constrain generalization. The benchmark remains relatively small, with 120 documents, 60 retrieval queries, and 150 STS pairs, and GTE still reaches perfect retrieval scores, indicating limited headroom for at least one model. The relevance labels and STS reference values are researcher-defined rather than independently adjudicated by multiple annotators. A stronger follow-up benchmark should therefore enlarge the candidate pool, introduce harder within-domain negatives and more varied query forms, obtain independent multi-annotator judgments with an appropriate inter-rater agreement analysis, and retain query-level outputs for validated paired significance testing. The standardized raw-text protocol also does not manually add model-specific query/passage prefixes or task instructions, and CPU profiling covers embedding generation in one CPU-only environment rather than complete end-to-end RAG latency.

This study demonstrates that benchmark design materially affects what an embedding evaluation can reveal. Strengthening an initially shallow Thai retrieval configuration substantially increases discrimination across ten open-source systems, while GTE Multilingual Base still reaches the ceiling of the controlled testbed. The result shows that broad structural saturation can be reduced without eliminating model-specific ceiling effects. The final live-model results expose distinct operating regions. GTE achieves the strongest observed retrieval and STS results; BGE-M3 and Multilingual-E5 Large remain closely matched strong performers; E5-Base, MPNet, LaBSE, and SEA-LION form another compact high-performing group; and WangchanBERTa plus the lightweight controls show larger reductions under the standardized raw-text protocol. Corrected tokenizer-based CPU throughput further demonstrates that retrieval effectiveness and inference efficiency do not align on a single axis. Future work should enlarge the Thai candidate collection, introduce harder within-domain negatives and more varied query formulations, and obtain independently adjudicated relevance and STS judgments from multiple annotators with reported inter-rater agreement. Query-level outputs should also be retained to support validated paired statistical tests for closely matched systems. Additional extensions include comparing the common raw-text protocol with model-specific recommended inference configurations and evaluating lexical or hybrid retrieval, reranking, vector-database latency, quantization, and end-to-end RAG workloads.

Generative AI and AI-assisted tools were used to support literature organization, Python scripting and debugging, English-language refinement, and Thai-English translation during manuscript preparation. All experiments, including model evaluation and throughput measurement, were executed and verified by the authors. The authors designed the study, supervised the methodology, interpreted the results, and take full responsibility for the accuracy, integrity, and final content of the manuscript.

The authors gratefully acknowledge the support of the Faculty of Business Administration, Maejo University, and the Computer Science Department, Faculty of Science, Chiang Mai University, which provided institutional and technical support for this research. The authors also thank the open-source communities behind PyThaiNLP, Hugging Face, sentence-transformers, BAAI, and AI Singapore for making the tools and pretrained models used in this study publicly available.