Evaluating Gemini and Gemma Language Models for Indonesian Text-to-SQL Tasks

Author

Galih Hermawan, S.Kom, M.T Dr. Ednawati Rainarli, S.Si., M.Si.

Abstrak

The Text-to-SQL project converts plain human language to SQL to make databases more accessible to everyone, but there are few Indonesian resources; this research provides an Indonesian Text-to-SQL benchmark and tests modern language models. We developed a benchmark of 60 queries across six complexity types on two databases (Sakila and a cooking Resep database) and tested four models - Gemini 2.5 Flash, Gemini 2.5 Flash Lite, Gemma-3-27B-IT, and the locally run Gemma-3n-E2B, with measures of execution success, row-match accuracy, speed (ms), and token usage. The model (Gemini 2.5 Flash) achieved 100% execution success and 95% row-match correctness, with an average response time of 4,707 ms. With a moderate accuracy drop to 75%, Flash Lite reduced the delay to 1,249 ms. With 5,684 ms, Gemma-3-27B-IT achieved 81.7-82% of its accuracy. With a latency of 159,818 ms, the CPU only Gemma-3n-E2B achieved 21.7-22% accuracy. Models hosted on the cloud are highly accurate at interactive delays, although small local models running on the CPU are not. We conclude that Gemini 2.5 Flash is the most reliable overall, while Flash Lite offers a practical speed-accuracy compromise. To support reproducibility and further research impact, we release the benchmark and evaluation pipeline and outline extensions (conversational contexts, retrieval-augmented prompting, quantization, and caching) to improve the viability of lightweight local deployments.