Tokenizing Numerical and Embedding Features for LLM RecSys
2026-07-10 • Information Retrieval
Information RetrievalMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs), which are good at handling text, are used in recommendation systems. They noticed that these models usually focus only on text data, missing out on numerical and other detailed features that are important for good recommendations. To fix this, the authors created a method to convert numbers and embeddings into a form LLMs can understand as if they were text tokens. Their experiments showed this approach improves how well the system can find relevant recommendations, especially when the combined features interact before being processed by the model.
Large Language ModelsRecommender SystemsSequence ModelingEmbedding FeaturesNumerical FeaturesTokenizationTwo-Tower Retrieval ModelFeature FusionInteraction Module
Authors
Zhe Xu, Ankit Peshin, Chiyu Zhang, Feng Qi, Johnson Lui, Anil Ramakrishna, Justin Johnson, Carl Hu, Kaushik Rangadurai, Luke Simon
Abstract
Large language models (LLMs) are increasingly used as backbone architectures for recommender systems because of their strong sequence modeling and representation learning capabilities. However, most LLM-based recommenders operate primarily on discrete textual tokens, whereas practical recommendation pipelines also rely on continuous numerical features and dense embedding features produced by upstream feature engineering or pretrained encoders. This mismatch limits the ability of LLM-based models to exploit fine-grained non-textual signals. We propose a soft-token fusion framework that maps numerical and embedding features into the LLM embedding space, allowing heterogeneous recommendation signals to be consumed through the standard token interface. We instantiate the framework in a shared-parameter LLM-based two-tower retrieval model and introduce an interaction-based fusion module that refines embedding and numerical soft tokens before they are inserted into the final LLM input. Experiments on three Amazon recommendation benchmarks show that soft-token fusion improves retrieval performance over LLM-based baselines, and that interaction-based fusion is more effective than direct concatenation of heterogeneous soft tokens.