Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry
2026-07-16 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explore a method to identify languages in text using character and bigram frequencies treated as compositional data, which respects their proportional nature. They use a mathematical transformation called centered log-ratio (CLR) to convert these frequencies into a form where regular distance calculations work properly. Their approach combines these transformed features with smoothing techniques to handle rare data and performs well across six languages, especially with longer texts. This method is efficient, interpretable, and needs fewer computational resources than typical neural network models.
language identificationneural architecturesn-gram modelscompositional datacentered log-ratio transformationAitchison distanceLaplace smoothingunigrambigramfrequency distributions
Authors
Paul-Andrei Pogăcean, Sanda-Maria Avram
Abstract
Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via the centered log-ratio (CLR) transformation bijectively onto the $(D-1)$-dimensional zero-sum subspace of $\mathbb{R}^D$, where Euclidean distances correspond to Aitchison distances. A pipeline is proposed, combining CLR-transformed unigram and bigram features with Laplace smoothing to address sparsity. The method is evaluated on six languages. Experimental results show that the proposed approach achieves robust accuracy across different text lengths, with strong performance for longer sequences. These findings indicate that compositional representations provide a deterministic and computationally efficient alternative for language identification, particularly in settings where interpretability and low resource consumption are essential.