Domain-specific small language models for creative text generation: a case study on the backrooms

Abstract

Large Language Models (LLMs) occupy a prominent place among Machine Learning models that have been revolutionizing modern life as an indispensable support tool in the most diverse areas of human activity. Nevertheless, the popularization of large LLMs faces a significant limiting challenge: the high cost associated with their use, whether through expensive hardware, cloud processing or paid APIs. In this light, the main objective of this work is to propose an approach to generate smaller language models, which can be trained in more modest hardware and that are effective in generating texts related to specific domains. The case study chosen for that are The Backrooms, an online legend which describes an alternative reality with different levels that are described with very unique, surreal and rich descriptions. For this, a Backrooms dataset, was created and used to produce two small language models (SLMs) to generate Backrooms level descriptions: Backrooms-Llama, conceived from the fine-tuning of the open source pretrained model Llama 1B; and Tiny Backrooms, a decoder-only architecture pretrained from scratch and then fine-tuned. The performance of these two models was evaluated against the famous and much larger DeepSeek-V3. The generated level descriptions of the three models were evaluated by using chatGPT 4o-mini as judge and human judges. Additionally, a brand new evaluation method, which consists on generating pictures with the output of each competitor model and, then, evaluating these pictures alongside their descriptions with chatGPT 4o, was proposed. The results found are very promising, showing that, even though DeepSeek performed better in general, Backrooms-Llama and Tiny Backrooms, operating in a much more modest architecture, also got mainly relevant evaluations and were able to learn how to generate Backrooms level descriptions. Such results indicate that, in creative generative tasks, a lower cost domain-specific SLM, when properly optimized, can achieve performance comparable to that of consecrated LLMs.