Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry
2026-08-18 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how to use small, in-house AI models to sort invoices into the right financial categories, which is usually a tricky task for accountants. They looked at how these models understand financial language and found that grouping invoice data by vendor helps the AI do a better job. Their fine-tuned model performed well even with limited data, beating some bigger models and simple methods. Interestingly, adding extra structure that helps humans didn't improve the AI's results. Overall, the authors suggest small, customized AI can effectively support invoice categorization while keeping data secure.
General Ledger (GL) codeinvoice categorisationsmall language models (SLMs)sentence transformersembedding geometrySBERTDeBERTafinancial corpusclassification accuracyfine-tuning
Authors
Emma Ceccherini, Daniel Lawson, Anjulika Salhan
Abstract
Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.