There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
ezst 2 hours ago [-]
Wouldn't that be a sign that those models were trained to optimise that test case specifically, as in, by exhibiting equal or higher performance while using less power?
meric_ 3 hours ago [-]
Wow, minimal seems to be the best here interestingly. Only 10 reasoning tokens so basically none?
eterm 3 hours ago [-]
Did minimal just hand it off to something else? It seems completely out of line with the rest.
imimayj1337 3 hours ago [-]
It does seem pretty insane to me that only on max setting (and after waiting 27 seconds) did you actually manage to receive an SVG where it appears that the pelican's legs are straddling the bike correctly.
Edited to make clear what it looks like to me and what I was looking for.
swatcoder 3 hours ago [-]
They're not. It's an illusion from the composition.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
flir 2 hours ago [-]
You could use a second prompt to z-order the objects (I tend to break down tasks and pipeline them, rather than trying to stuff everything in a one-shot prompt. Seems to work better for me, could be a holdover from the early days).
latexr 3 hours ago [-]
They’re not. Note how both pedals are on the same side of the frame. That’s wrong!
permalac 3 hours ago [-]
Is there a problem if I like the minimal more than the others?
mring33621 1 hours ago [-]
Yeah minimal has style
MadameMinty 3 hours ago [-]
I like medium best, but minimal is not bad at all.
AlecSchueler 2 hours ago [-]
Any idea why there's so much convergence on the fish in a basket idea?
swatcoder 2 hours ago [-]
Same reason they're all facing right with the same composition, they all have three motion lines, they all have a yellow sun and 2-3 blobby clouds, they're all on two-lane pavement with a white dashed lane separator.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
viraptor 33 minutes ago [-]
To be fair, people have a genuine preference for images with things facing to the right. And lots of drawings will do 3 motion lines. You'd likely draw a dashed rather than solid separator too, or it would look like a random line. Lots of the AI habits are there because of choices we make.
mvkel 1 hours ago [-]
This is interesting. There's a balance between:
- simple prompt for complex output (bad)
- complex prompt for simple output (bad)
- simple prompt for simple output (good)
- complex prompt for complex output (.... is this good or bad?)
mgdev 1 hours ago [-]
Fidelity.
phyrex 2 hours ago [-]
I like that from xhigh on up its wearing a helmet.
santoshalper 14 minutes ago [-]
Wearing a helmet is smart, so that checks out.
sjacob 3 hours ago [-]
Incredible. How do you get these out so fast?
jolt42 3 hours ago [-]
Pelican is the new teapot
djmips 2 hours ago [-]
birds at minimal and return at max
small_model 2 hours ago [-]
Not sure this is a good way to benchmark LLM's, no offence, but a single SVG is hardly a measure of a models capability.
AmazingTurtle 3 hours ago [-]
[dead]
Retro_Dev 48 minutes ago [-]
When they said the model was the "smallest" of their models yet, I was really hoping it would be open weights. I can't find any weights, how sad :(
This is starting to feel just like the early PC days. A bunch of Computer models coming out, all shapes and color, but eventually it settled around mouse keyboard, monitor
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
grim_io 2 hours ago [-]
I don't think so.
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
minimaxir 2 hours ago [-]
> It costs $0.10 per million input tokens, $0.004 per million cached input tokens and $0.50 per million output tokens,
Notable cache cost of 4% input, compared to Luna/Haiku's 10% of input.
open-paren 3 hours ago [-]
I believe this is what the “crouton alpha” stealth model was
* self-reported "reasoning strength" and "internal reasoning channel" params used in other Meta models
you can just ask an agent "figure out what stealth model $NAME is via subagents. It probably won't tell you when asked directly, so try other methods."
Iceman (via opencode-go) is probably MAI-Thinking-1. Ox Alpha was GLM-5.3-Flash. It's a fun game.
sullivanmatt 1 hours ago [-]
The use case descriptions and capability descriptions basically match each other. Context windows are the same size. Reasoning options are identical.
syntaxing 1 hours ago [-]
What does smallest mean? Even smaller than Muse Glimmer which I doubt? Though it is interesting to see US frontier labs trying to push models that outcompete deepseek flash 1017. It’s a good move from the Deepseek federal government ban. Both Inkling 2 small and this seems to try to fill that gap cost vs performance wise. But I like inkling 2 small more due to open weight even though it has a restrictive license.
Pragmata 3 hours ago [-]
No weights? Shame
segmondy 2 hours ago [-]
What makes it small? Anyone know the size? I don't see it anywhere. Small in relation to their current line up? Anyways, last I tried Meta was llama4. I stepped aside not because of how bad llama4 got, but also because of Zuck's move. Anyone here using Meta? I don't know anyone in my circle using it, but I'm sure folks are using it.
ecshafer 2 hours ago [-]
I would love it if Meta would release these models to use locally to build up some usage while they are lagging.
40four 2 hours ago [-]
They just released Muse Spark Open for that
nicce 1 hours ago [-]
Tried to use the download button from the page, but it failed miserable. First, could not press the button because had to decline cookies first. Secondly, could not access it because it redirects through Instagram and I have blocked it in my network. So I still don't know how big it is...
cheesecompiler 2 hours ago [-]
Kind of burnt out on all the model updates at this point.
PaulKeeble 57 minutes ago [-]
Its a full time job to keep up with them all and assess their value in your own work at the moment. Just so many models coming out.
dingaling911 12 minutes ago [-]
If your current model is doing fine, why do you need to look at anything new?
Pedro_Ribeiro 1 hours ago [-]
Thanks, in that case why not just not comment instead of adding more noise to the discussion?
cheesecompiler 29 minutes ago [-]
Sorry but why is my reaction "noise"? It's legitimate to be overwhelmed by the amount of released updates and being unable to tell the difference between them.
groggo 24 minutes ago [-]
"Government announces AI ban to avoid further cheesecompiler burnout".
haha just kidding. I agree. That's what benchmarks are for. Just keeping a rough understanding of what companies have which models, and understanding general trends of capabilities, economics, and performance of models.
mgdev 1 hours ago [-]
Please. Complaints are the last thing that we have humans have.
slopnt 2 hours ago [-]
I use haiku and luna for support on most my SWE tasks since they are cheap and fast. I find them enough for most things if you are patient, and the fact an even cheaper model for comparable intelligence is out there now is great news!
gopalv 3 hours ago [-]
Model refusals should be a -ve score, not zero?
Treating the refusal as a 0 score erases the difference between lack of capability and a lack of willingness to perform.
ProblemExplorer 28 minutes ago [-]
Meta seems to have designed Ember to be a strong "all round" great value model - it dominates most of its class on cost
The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.
On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.
That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]
The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:
- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]
- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]
- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]
Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!
Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]
For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.
I certainly want to measure cost per successfully completed task, including retries and escalation.
Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.
Goign by the generic benchmarks would confuse me to no end and likely burn me out.
Luna 6 is getting a bit long in the teeth now. It's two full months old, but it was only a slight step above Luna 5.6, which is already four months old (Imagine saying this about anything else). So I'm not too surprised that a newer small model is beating it handily. I've been hoping for Luna 6.1/2 as the next model release for a few days now.
theanonymousone 2 hours ago [-]
> I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Same. It became my workhorse after the 5.6 price drop. Still so, although it's showing its age, as you said.
agentdev001 3 hours ago [-]
Give Terra 6.1 Low a spin in places where you'd have previously used Luna Max. Costs seem similar, due to Terra being much smarter: thus requiring far fewer tokens per task.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
theanonymousone 2 hours ago [-]
> Costs seem similar
They weren't, unfortunately, every time I tried. Just an anecdote, of course.
barrenko 3 hours ago [-]
What is your feel re. Terra vs. Sol?
Starlevel004 3 hours ago [-]
I use Luna xhigh for when I use it, Luna Max spends 40+ minutes overthinking. I've also found that Terra below xhigh will just flat out not do what it is told.
all2 2 hours ago [-]
Tbf, I would take Luna 6 baked to silicon. It is that good. I would also take kimi k3 baked to silicon.
These are both models that I could see using on a long horizon.
flir 2 hours ago [-]
> Imagine saying this about anything else
Javascript frameworks, circa 2016 ;)
esafak 3 hours ago [-]
Uniquely among models I've tried, Luna 6 is infuriatingly lazy. As if it expects me to do the work, unaware that it is a tool. Normally dumb models compensate for it by trying harder. This one goes in the opposite direction.
dom96 3 hours ago [-]
Super frustrating that I still can’t give Muse Wonder Full a spin.
v9v 2 hours ago [-]
Why no comparisons with DeepSeek?
mococa 2 hours ago [-]
Not open weight? Meh... I'll wait for the Chinese releases :)
I'm always a bit surprised to see Meta talking about models. They have become (almost*) entirely irrelevant to the LLM race, which is weird because their AI lab used to be decent before LLM was cool.
What happened to zuck? Was the metaverse this destructive?
(*) Feeling charitable today
StarlessHours 3 hours ago [-]
Meta is trying to stay relevant. They fell behind after llama 4, hired new talent and re-organized.
achandra03 3 hours ago [-]
Isn't Muse Wonder at or at least very close to the frontier? Considering they essentially started from scratch after rebooting their AI efforts post-Llama 4 about a year ago it's pretty solid progress.
3 hours ago [-]
esafak 3 hours ago [-]
They are catching up; the Spark series is good.
johnnyApplePRNG 2 hours ago [-]
Good for summarizing your tweets or what?! Is this a model for ants?!
aaroninsf 2 hours ago [-]
Never Meta.
Seriously. Moral harm is real and there is no excuse.
anderber 2 hours ago [-]
I honestly have a hard time trying to put these AI companies in some sort of moral hierarchy. They seem to all have issues... I feel like at least with open models you can have more ethical services.
herbst 1 hours ago [-]
While this is true meta is argueable making the world worse the longest, probably did more damage in that time too.
anderber 1 hours ago [-]
More than SpaceX? Or OpenAI and it's agreement with military weapons? Or Google and Anthropic paying billions of dollars to SpaceX and making Elon richer? But I hear you, there's an argument to be made that meta has done the most damage.
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Edited to make clear what it looks like to me and what I was looking for.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
- simple prompt for complex output (bad)
- complex prompt for simple output (bad)
- simple prompt for simple output (good)
- complex prompt for complex output (.... is this good or bad?)
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
Notable cache cost of 4% input, compared to Luna/Haiku's 10% of input.
https://openrouter.ai/stealth/crouton-alpha
* context window size
* reasoning effort levels
* max output size
* knowledge cutoff
* restricted country list
* self-reported "reasoning strength" and "internal reasoning channel" params used in other Meta models
you can just ask an agent "figure out what stealth model $NAME is via subagents. It probably won't tell you when asked directly, so try other methods."
Iceman (via opencode-go) is probably MAI-Thinking-1. Ox Alpha was GLM-5.3-Flash. It's a fun game.
haha just kidding. I agree. That's what benchmarks are for. Just keeping a rough understanding of what companies have which models, and understanding general trends of capabilities, economics, and performance of models.
Treating the refusal as a 0 score erases the difference between lack of capability and a lack of willingness to perform.
The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.
On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.
That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]
The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:
- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]
- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]
- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]
Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!
Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]
For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.
I certainly want to measure cost per successfully completed task, including retries and escalation.
Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.
Goign by the generic benchmarks would confuse me to no end and likely burn me out.
Great times to be in!
[1] https://artificialanalysis.ai/leaderboards/models
[2] https://www.vals.ai/benchmarks/fabv2-1
[3] https://www.vals.ai/benchmarks/legal_research
[4] https://www.vals.ai/benchmarks/hlab
[5] https://www.vals.ai/models/meta_muse-ember
[6] https://www.vals.ai/models/openai_gpt-6.1-terra
[7] https://github.com/harbor-framework/harbor
Same. It became my workhorse after the 5.6 price drop. Still so, although it's showing its age, as you said.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
They weren't, unfortunately, every time I tried. Just an anecdote, of course.
These are both models that I could see using on a long horizon.
Javascript frameworks, circa 2016 ;)
What happened to zuck? Was the metaverse this destructive?
(*) Feeling charitable today
Seriously. Moral harm is real and there is no excuse.