GPU-Accelerated Gradient-Based Space-Filling Design with an Application to Systematic Evaluation of Large Language Models on Financial-Regulatory Documents

Abstract

Large language models (LLMs) and agentic AI systems are increasingly deployed in high-stakes settings such as financial regulation, where accuracy must be evaluated not merely on average but conditional on the many input factors that shape each query: question phrasing, user persona, reasoning demands, document context, adversarial pressure, and the question itself. Such evaluation is not benchmarking but designed experimentation, with a binary correctness response and a high-dimensional, mixed-cardinality factor space. While most empirical studies of LLMs report aggregate accuracy, attributing failures to specific factor levels demands designs that maximize statistical power across the joint factor space. However, classical space-filling tools—Latin hypercube optimization, the Enhanced Stochastic Evolutionary algorithm, and quasi-random Sobol sequences—either scale poorly to the regimes required by modern LLM evaluation campaigns or do not optimize the geometric coverage that drives statistical power. In this article, we develop a GPU-accelerated, gradient-based methodology for generating space-filling designs in continuous [0, 1]^m space, built on (i) an analytic gradient of the φ_p criterion computed entirely in log-space and reaching the O(n²) memory floor; (ii) a sigmoid reparametrization with p-annealing; (iii) a hybrid Sobol+Refine initialization; and (iv) a cardinality-weighted distance metric for mixed-cardinality factors. We connect the φ_p criterion to the noncentrality parameter of likelihood-ratio tests for factor effects via fill-distance bounds, formalizing the route from design geometry to statistical power, and we show that the cardinality weighting is optimal under a projection-balance criterion. We then apply the methodology to FinStructBench, a benchmark we develop for evaluating LLMs on structured information retrieval from financial-regulatory documents, with graph-verifiable ground truth. In an 880-run experiment with 21 mixed-cardinality factors (cardinalities 2–44), the optimized design yields a logistic model with substantially better fit (ΔAIC = −90.5, McFadden R² 0.752 → 0.825) and detects factor effects in 7 of 9 question categories versus 4 for the Sobol baseline—a more informative diagnostic of the model’s failure modes from the same number of API calls.