BenchmarksCommunityAI

How we benchmark LLMs from r/LocalLLaMA chatter

2 min
On this page4

If you have opened the comparison page, you have seen eight models — GLM 5.3 Flash, Kimi K3, DeepSeek V4.1 Flash, and friends — each scored on ten categories with an overall number on top. Where do those numbers come from? Not from vendor APIs, not from submitted benchmark suites, and not from our own private evals. They come from the community itself.

We collect the top posts and comments from r/LocalLLaMA over the past month and read what real users said about each model. A single model's sample is big: GLM 5.3 Flash alone spans 10 posts and 398 comments. That means the signal behind every score comes from hundreds of distinct voices, not from one influencer thread or a launch-day hype cycle.

Most reports come from r/LocalLLaMA, and exactly one is built the same way from another community — Claude Opus 5.5, read from r/singularity — with every report labelled with the subreddit it was drawn from.


From chatter to category scores

Each model gets ten scores from 0 to 100: Intelligence, Speed, Coding quality, Instruction following, Long context, Price / value, Reliability & uptime, Creativity & writing, Multimodal & vision, and Agentic & tool use. Every score carries a written rationale tied to what the community actually said — which users praised its speed, which ones hit refusals, what people happily pay for it. Each report also adds an overall score, pros and cons, top quotes with links back to the source posts, and a confidence level derived from the sample size, so you can see how heavily to weight the verdict.

No data beats a guessed number

This is the part we care about most. When the sample doesn't discuss a capability, the score is No data — never an interpolated guess. If nobody talked about a model's multimodal vision, we don't invent a number to make the chart look complete. On the comparison page, models with missing categories simply get incomplete radar shapes instead of faked symmetry. It looks less tidy, and it's honest.

Reading the comparison page

The comparison page shows all ten axes at once so you can spot trade-offs at a glance — a model that codes well but costs a fortune stands out immediately. Per-category bar charts show the spread between models when you want detail, and per-model toggles let you include or exclude models from the view. Every model also has its own report page underneath, where the rationales, pros and cons, and top quotes live.


Takeaways

  1. Community-only signal. Scores come from r/LocalLLaMA posts and comments from the past month (r/singularity for Claude Opus 5.5) — no vendor APIs and no submitted benchmarks.
  2. Ten categories, each with a rationale. Every 0–100 score is tied to what users actually said, plus an overall score, pros and cons, and top quotes.
  3. Gaps stay gaps. Undiscussed capabilities read "No data," and comparison radars render incomplete shapes instead of invented numbers.
  4. Built for scanning. All ten axes at once, per-category bars, and per-model toggles — with a full report page behind each model.