Why small language models are the future of AI

What a small language model is

A language model is the part of an AI that reads your question and writes the answer. The large ones, like the models behind ChatGPT and Gemini, run in data centres. A small language model, or SLM, has a few billion parameters instead of hundreds of billions. That’s small enough for a laptop or a phone.

There’s no official cut-off. Researchers at NVIDIA call a model small when it fits on a normal consumer device and answers fast enough for one person (NVIDIA Research). In practice that’s usually under 10 billion parameters. The model Apple puts on your iPhone has about 3 billion (Apple).

Small models got good fast

In 2022, the smallest model that scored above 60% on MMLU, a large test of general knowledge, was Google’s PaLM. It had 540 billion parameters. In 2024, Microsoft’s Phi-3-mini passed the same bar with 3.8 billion. That’s 142 times smaller in two years (Stanford AI Index 2025).

The price fell even faster. An answer at the level of GPT-3.5 cost about $20 per million tokens in November 2022 and $0.07 in October 2024. That’s more than 280 times cheaper (Stanford AI Index 2025).

And the gap keeps closing. Epoch AI found that open models that run on a single gaming graphics card match the best AI from 6 to 12 months earlier (Epoch AI).

Why small wins

Most jobs are small jobs

Putting the dentist in your calendar, summing up an email, checking whether a text is a scam. You don’t need the largest model in the world for that. The NVIDIA researchers make this case for AI agents: most steps an agent takes are repetitive and narrow, and a small model handles them well. They estimate that running a model of 7 billion parameters is 10 to 30 times cheaper than one of 70 to 175 billion, in speed, energy and computing power (NVIDIA Research).

Companies see it too. Gartner expects organisations to use small models built for one task at least three times as much as general large models by 2027 (Gartner).

It runs on what you already have

The IEA expects data centres to use more than twice as much electricity by 2030, around 945 TWh. That’s slightly more than all of Japan uses today (IEA). A small model on your phone runs on a chip that’s already in your pocket. It also works without a connection.

It costs nothing per question

Every question to a large model in the cloud costs the company money, and someone pays for that: with a subscription, with ads or with your data. Apple’s on-device model has no cost per question for the apps that use it (Cult of Mac). So an app can give its AI away for free.

Your question stays with you

When the model runs on your device, your question doesn’t have to go anywhere. No company stores it, reads it or trains on it. For questions about your health, your money or the people around you, that matters most.

Melo answers on your own iPhone and Mac. Your questions don’t go to Kamelo or any other AI company. Get Melo

What small models can’t do yet

• They know less. A few billion parameters can’t hold every fact a huge model knows, so on hard knowledge questions they get more wrong. Looking things up helps.
• Long, hard thinking, like a difficult proof or a large piece of code, still goes better on large models.
• They need a recent device. Apple Intelligence needs an iPhone 15 Pro or newer.

So the future is mixed. A small model on your device does most of the work, and only the hard question goes to a bigger model. The NVIDIA researchers propose the same: systems with models of different sizes, with the small one as the default (NVIDIA Research).

How Melo uses small models

Melo is built on this idea. On iPhone, Melo answers with the model that Apple Intelligence puts on your phone. On Mac you can also choose Gemma 4, a model that runs on your Mac. Your questions don’t go to Kamelo or to any other AI company.

With that, Melo plans your day, finds recipes, reads your mail and warns you about scam texts. When Melo looks something up, only the search words leave your device, never your chat.