Abacus.AI Open Source | Open Source AI & LLM Models

Today our focus is fine-tuning frontier open-weight models into reliable AI agents. The refreshed Smaug line applies one methodology — human-curated, real-world agentic traces combined with synthetic data grounded in hard examples — to three open-weight bases: Smaug Flash on DeepSeek V4 Flash for always-on enterprise agents, Smaug Mini on Qwen3.8 27B for compact multimodal tasks, and Smaug Agentic on Kimi K3 at frontier scale. We release the strongest results back to the community on Hugging Face.

Our open-source contributions to LLMs have led to several other open-source labs adopting some of our techniques and pushing the boundaries of enterprise and SOTA AI.

The Smaug Line

Here are the key contributions from Abacus.AI to open-source:

Smaug Flash

Smaug Flash is an agentic fine-tune of DeepSeek V4 Flash and the workhorse of the line. It is tuned for continuously running enterprise self-improving agents: loops that stay alive for hours, carry a long context from step to step, and spend most of their tool calls outside code — reading and writing documents, querying data systems, calling APIs, and driving automations. DeepSeek V4 Flash is fast, cheap and robust for frequently running agentic tasks but is susceptible to spins and confusion in long-context tool use; Smaug Flash was trained to make those loops faster and less prone to spins and stalls at maximum reasoning effort while keeping the cost and speed of the base.

Only the attention factor matrices are adapted — three LoRA adapters merged as full deltas — so the weights load exactly like the official release, with the same 1M-token context, and any serving stack that runs DeepSeek V4 Flash runs Smaug Flash unmodified. On our paired runs it gains +14.3 on LiveBench agentic coding, +13.7 on AutomationBench and +19 on NL2Repo-Bench over the base, and leads Claude Sonnet 5 on every benchmark we compared. Full results are on the research page and in the model card on Hugging Face.

Benchmarks Smaug Flash vs DeepSeek V4 Flash (base)

LiveBench (Overall) Smaug Flash: 77.4 DeepSeek V4 Flash: 74.2
LiveBench Agentic Coding Smaug Flash: 61.1 DeepSeek V4 Flash: 46.8
DeepSWE 1.1 Smaug Flash: 56.6 DeepSeek V4 Flash: 54.4
NL2Repo-Bench Smaug Flash: 73.3 DeepSeek V4 Flash: 54.2
AutomationBench Smaug Flash: 38.8 DeepSeek V4 Flash: 25.1

Higher is better. Base DeepSWE, NL2Repo-Bench and AutomationBench scores are vendor-reported; the rest were run by Abacus.AI under the same harness.

Smaug Agentic

Smaug Agentic is the largest model in the line and the clearest demonstration of how the recipe scales: an agentic supervised fine-tune of Kimi K3, the frontier Mixture-of-Experts model from Moonshot AI, trained on filtered multi-turn, tool-using coding trajectories with reasoning tokens masked from the loss. Every architectural parameter is unchanged from the base, so any inference stack that serves Kimi K3 serves Smaug Agentic as a drop-in replacement, and interleaved thinking is preserved across turns.

On our own runs it improves on the published Kimi K3 figures on GPQA Diamond (94.1 vs 93.5), DeepSWE (69.9 vs 67.5), SciCode (60.8 vs 58.7) and LiveBench Agentic Coding (64.6 vs 62.2). The behavioral change is bigger than the deltas: runaway reasoning is suppressed while normal deliberation is preserved, and across 113 agentic coding tasks and more than seven hours of continuous work it ran a median of 78 agent steps per task with no infrastructure errors and no timeouts. Full per-benchmark notes are in the model card on Hugging Face.

Benchmarks Smaug Agentic vs Kimi K3 (base)

GPQA Diamond Smaug Agentic: 94.1 Kimi K3: 93.5
AA-LCR Smaug Agentic: 75.7 Kimi K3: 74.7
DeepSWE Smaug Agentic: 69.9 Kimi K3: 67.5
Terminal-Bench 2.1 Smaug Agentic: 86.5 Kimi K3: 88.3
SciCode Smaug Agentic: 60.8 Kimi K3: 58.7
LiveBench Agentic Coding Smaug Agentic: 64.6 Kimi K3: 62.2
AutomationBench Smaug Agentic: 31.0 Kimi K3: 30.8
MMMU-Pro Smaug Agentic: 81.0 Kimi K3: 81.6

Higher is better. Kimi K3 scores are as published on its model card; Terminal-Bench 2.1 uses different agents for the two models.

Smaug Mini

Smaug Mini is an agentic fine-tune of Qwen3.8 27B, the compact member of the line. It targets multimodal use cases and smaller reasoning tasks — the one-off jobs that need to look at an image or a video, follow a precise instruction and call a few tools — and inherits the base's native image and video input. Applying the same methodology to a 27B dense model produces the same shape of result as on the larger bases: on our runs Smaug Mini leads Claude Sonnet 5, Claude Opus 4.6 and GPT-5.6 Luna on IFBench, AutomationBench and JobBench, at a size that fits on a single GPU.

Benchmarks Smaug Mini vs Qwen3.8 27B (base)

LiveBench (Overall) Smaug Mini: 76.9 Qwen3.8 27B: 75.3
IFBench Smaug Mini: 82.0 Qwen3.8 27B: 79.5
LiveBench Agentic Coding Smaug Mini: 60.8 Qwen3.8 27B: 61.4
NL2Repo-Bench Smaug Mini: 55.8 Qwen3.8 27B: 42.3
AutomationBench Smaug Mini: 41.8 Qwen3.8 27B: 37.3
JobBench Smaug Mini: 50.5 Qwen3.8 27B: 33.4

Higher is better. LiveBench rows are the published livebench.ai 2026-06-25 leaderboard entries; the rest were run by Abacus.AI under the same harness.

Our Benchmarks

LiveBench AI

With LLMs training on web-scale data, test-set contamination is a pervasive concern in LLM evaluation that can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and they are not reliable for hard questions. For example, LLM judges make mistakes up to 40% of the time on challenging math and reasoning tasks.

To resolve this issue, we developed LiveBench, a contamination-limited benchmark that evaluates large language models on a variety of general intelligence capabilities including reasoning, coding, language understanding, data analysis, instruction following, and mathematics. By frequently releasing updated question sets and developing new tasks over time, we ensure that our results remain an accurate assessment of LLM capabilities as new models are released. Questions are constructed from a variety of recent sources such as research papers and news articles so they can be easily refreshed over time. LiveBench also uses only objective ground-truth judgment to ensure unbiased results.

Our leaderboard at livebench.ai presents results for all major model providers, both open source and proprietary. Since its release, LiveBench has remained one of the most popular LLM benchmarks and has been a featured result in major model release reports.