I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
matthewdgreen 2 hours ago [-]
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost.
Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
Groxx 4 hours ago [-]
Fair (I haven't been using the encrypted-reasoning systems, though this is common in open ones - I'm kinda surprised it's an option in encrypted ones too), though what they're doing here is cross-user replays in addition to cross-model.
myworkaccount2 4 hours ago [-]
There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves?
This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.
If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.
For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
supriyo-biswas 3 hours ago [-]
It’s about being able to change models mid-task. For example, I want to be able to plan using Fable but implement the plan using Sonnet, and that won’t work if this is implemented.
crabette 1 hours ago [-]
For plan it's relatively easy, just make the plan the artifact. The point is to ingest knowledge with one model and use it in another, and that is not necessarily easily expressible in natural language.
SaltyBackendGuy 2 hours ago [-]
Or even my fable credits run out mid task and need to switch back to opus >.<
QuercusMax 2 hours ago [-]
Prior to LLMs I never considered that I might have to make a resource-usage decision between hiring Star Trek's Data vs. his stupider brother B4...
100% guaranteed that this research just forced this to happen now.
Sucks.
dannyw 2 hours ago [-]
It’s already patched according to the authors. Details were not specified.
pas 3 hours ago [-]
or add some metadata and don't allow downgrading.
aix1 4 hours ago [-]
I really don't understand why server-side storage of the trace isn't a viable approach here, with only a unique key flowing to the client and back. Does it have something to do with how backend load-balancing works?
sandeepkd 4 hours ago [-]
Yes, this storage would be growing exponentially making the disk space and latency problems harder (add the disaster recovery/backups). I think the choice of using client side is not too bad if you ensure that its secured properly. Also the company can excuse itself from the liability of storing sensitive data on its servers, thats a big deal in itself to be compliant for enterprise audits
1. The down side is that it cannot be used across the clients even for the same user
2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
aix1 3 hours ago [-]
Having thought about this a little more, it's clear that server-side storage is not compatible with Zero Data Retention (ZDR). However, in non-ZDR settings, it seems likely that the providers are capturing all that data anyway?
> a per user key would have solved this issue for sure
It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
sandeepkd 3 hours ago [-]
Per user encryption key ties it with the user session (assuming you do authentication properly), no one else can access it. User being able to see the information is not really an attack vector in this case.
The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
amluto 3 hours ago [-]
Makes no difference. There is a policy as to whether to allow use of a reasoning trace in a given context. Whether that trace originates from authenticated ciphertext or a backend database is basically irrelevant.
aix1 3 hours ago [-]
Good point, thanks.
dannyw 3 hours ago [-]
That’s incompatible with zero data retention and so you’ll lose a lot of enterprise customers.
sandeepkd 4 hours ago [-]
I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.
The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.
1. Its a security issue.
2. Publicly available sessions make it much worse
cavisne 50 minutes ago [-]
You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.
sly010 16 minutes ago [-]
"Recovery" would be a more apt (although less catchy name). The stealing is on the provider side for not giving you access to tokens you already paid for.
vhantz 3 hours ago [-]
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
Aurornis 6 minutes ago [-]
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
AbhinavX 1 hours ago [-]
Not surprised. On many benchmarks (i.e tau), we have seen the same thing. Probably lots of training on every publicly available benchmark
throwa356262 3 hours ago [-]
Didn't we see this with Fable 5 on multiple benchmarks?
niemandhier 3 hours ago [-]
You cannot steal what is not owned.
At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
Perseids 1 hours ago [-]
I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
iamniels 2 hours ago [-]
On top of that, the user paid for those tokens, so if there is an owner, it should be the user, not the provider.
Zambyte 3 hours ago [-]
Even copyrighted information can never be "stolen". It can only copied without authorization.
otterley 2 hours ago [-]
Stealing is not a word that applies only to physical objects.
Zambyte 5 minutes ago [-]
Please provide an example of an action legally considered "stealing" that does not involve stealing physical objects.
margalabargala 45 minutes ago [-]
Any word can be applied to any concept with any meaning thanks to the fluidity of vernacular.
Language is all just sounds and markings. Anything can be redefined to mean anything, and anyone can decide to aggressively assert their preferred definition of a word.
kube-system 18 minutes ago [-]
Words can change meaning, but whether or not they actually do is a social function of how they are observably used.
Words can also have multiple meanings concurrently. When it comes to the word "steal" in these discussions, people generally are arguing past each other in regards to the single definition they're thinking of.
Of course, you can assert that the meaning of "steal" only applies to physical items. You are well within your right to do so. You'd be wrong, but you can do it.
otterley 16 minutes ago [-]
> people generally are arguing past each other in regards to the single definition they're thinking of.
It's more than that. By claiming that copyright infringement isn't stealing, they're usually doing so to justify such behavior: if the original thing remains with the owner, it couldn't have harmed him, could it?
kube-system 10 minutes ago [-]
Maybe but I try to give people the benefit of the doubt, especially when the argument at face value is factually incorrect.
niemandhier 47 minutes ago [-]
Funnily enough in some legal systems it does. Where I live the legal definition of “theft” is: Taking away a movable thing.
kube-system 32 minutes ago [-]
That's also... a different word.
kube-system 34 minutes ago [-]
Stole the words right out of my mouth!
Terr_ 2 hours ago [-]
While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.
For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."
8note 1 hours ago [-]
Im not so worried about that. Im worried about somebody impersonating me or doing something bad and not authorized with the cookies
cyanydeez 3 hours ago [-]
they also can't claim to be the owner of the output either because they can't claim to have had a valid license for the inputs.
twigger2 3 hours ago [-]
[dead]
andai 3 hours ago [-]
If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.
dgellow 2 hours ago [-]
Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches
HarHarVeryFunny 2 hours ago [-]
For all of the years of research, thinking and talking about model alignment, safety, confinement, etc, when it comes down to it these companies appear to be entirely incompetent.
Pragmata 22 minutes ago [-]
Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool...
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
I can’t believe they don’t validate a decrypted signature belongs to the user or use a unique encryption key per user/session.
cush 4 minutes ago [-]
I really like this website
pradeep1177 25 minutes ago [-]
These logs containing opaque blobs could accidentally contain secrets, the researchers decoded many of reasoning blocks from public repositories and reported finding PII and credentials.
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
SwellJoe 4 hours ago [-]
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
dannyw 2 hours ago [-]
Yeah, it’s also useful for prompt tuning, debugging and understanding how a model interprets your prompt.
Also really good for identifying any contradictions in your system prompt and context.
sm-silversight 36 minutes ago [-]
Is this basically a paper on how to distill, in exactly the fashion openai/anthropic don't want/say is copyright theft?
arjie 53 minutes ago [-]
Wow, almost certainly the approach that alternative labs use to distill Claude. I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.
Havoc 13 minutes ago [-]
TIL it actually sends the traces. I had assumed this is entirely server side
vinaigrette 4 hours ago [-]
I must say right of the bat this is the best research paper/working paper in regards to its styling. Beautiful
3 hours ago [-]
SwellJoe 4 hours ago [-]
I agree on desktop/laptop, but on mobile there are images that appear under the text making it hard to read.
user43928 3 hours ago [-]
On an iPhone Pro Max only the first trace is readable.
Navigating to the right lands between two cards, so that neither is readable.
thefourthchime 2 hours ago [-]
I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
EagleEdge 2 hours ago [-]
I used to do a very coarse version of this stealing. I ask a question from ChatGPT pro, once it is done, I ask claude chrome add-in to go through all those thinking from the side bar, extract everything along with all the sources used. Then try to reverse engineer the solution it came up with.
nervai 4 hours ago [-]
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace:
How to Steal Reasoning Without Reasoning Traces
https://arxiv.org/pdf/2603.07267
dannyw 2 hours ago [-]
Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.
x312 4 hours ago [-]
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
flexagoon 3 hours ago [-]
> I'm surprised these companies re-use the same encryption key across models
I assume switching the model in the middle of a conversation is intended behavior (very useful in coding agents, for example)
yubblegum 4 hours ago [-]
Seriously, what does it take to encrypt per session? There are many ways to make it scalable and efficient so I am wondering if this is left like this to allow interested 3rd parties ahem unobtrusively peek what people are doing with the AI.
(Thanks for the link. That’s an interesting idea!)
dannyw 2 hours ago [-]
The provider has the hidden text anyway; this isn’t customer managed encryption.
yubblegum 1 hours ago [-]
Sure, but if each session has a unique key then these need to be managed and stored and unauthorized access to these leaves tracks. So all that had to be 'compromised' is a single universally applicable key. Again, the question stands: session based encryption can be scalable and efficient. Why aren't they using it?
iamcoder18 4 hours ago [-]
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
kgeist 3 hours ago [-]
In the BlackHat presentation on the HuggingFace incident, OpenAI showed some excerpts from the reasoning traces, and they had that grug speak too (skipped articles, etc.). So the OP's method must have indeed found the actual reasoning traces.
lukewarm707 2 hours ago [-]
their gpt-oss models do the same. i don't use closed models so i never thought much about it.
gaigalas 3 hours ago [-]
Muse clearly does it to some extent. Saw a lot of that running Glimmer locally.
C0ldSmi1e 45 minutes ago [-]
Why they use different models to decode the reasoning content? Can the the model decode it?
aszen 44 minutes ago [-]
Because stronger models are harder to jailbreak from the paper it says haiku was easily fooled into giving us thinking contents
Cynddl 4 hours ago [-]
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
tanh 3 hours ago [-]
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
redox99 2 hours ago [-]
It wouldn't matter if it was stored only on their servers. As long as they offer the feature to downgrade a chat to a dumber model that can be jailbroken (and the downgrade keeps the reasoning), this trick works.
8note 58 minutes ago [-]
this is a lethal trifecta, but where a chunk isn't even needed
you have a secret to keep that is read by the llm, and untrusted input that wants to exfiltrate it.
by hell or high water, the agent is gonna output that text
lukewarm707 2 hours ago [-]
strange because, their subscriptions are not stateless. they log everything and send it to 3rd parties for moderation.
drob518 1 hours ago [-]
It’s scary the number of security tokens that end up being ingested by these models.
myworkaccount2 4 hours ago [-]
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
NitpickLawyer 4 hours ago [-]
> But the real question is: Is it okay to steal
By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)
I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.
(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)
desterothx 3 hours ago [-]
I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on
rfoo 1 hours ago [-]
Let's face it. Chinese labs made some of the biggest advancements. AND training on Claude (or GPT) output IS unreasonably effective. The two sentences are true at the same time.
tuesdaynight 4 hours ago [-]
It's like sideloading. It's very hard to fight against the marketing budget of big tech
orbital-decay 3 hours ago [-]
Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.
azinman2 4 hours ago [-]
The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.
pyrale 3 hours ago [-]
Stuff available on the internet is also the result of a lot of research, time, money, and expertise. And AI companies taught us that it’s OK to yoink whatever is not bolted to the ground, even when it is illegal to do so.
tristanj 4 hours ago [-]
No. There are dozens of companies that resell tokens at a discount to collect and resell session data to various Chinese labs.
elzbardico 3 hours ago [-]
Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers.
Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it.
Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests.
About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it.
Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
I understand it’s cool to have an artistic website, but it’s very noisy and non-accessible.
But very interesting result.
fractorial 5 hours ago [-]
Fascinating approach; however, a nightmare to scroll on mobile.
elzbardico 3 hours ago [-]
OpenAI and Anthropic will probably now resort to save this server side, instead of relying on encription to be able to keep state on the client.
khalic 4 hours ago [-]
This is beautiful work, congrats
bonoboTP 3 hours ago [-]
It's not stealing.
throwa356262 3 hours ago [-]
This is laughable security. People claim security is now "solved" thanks to AI but from where I am standings it looks more like the fun 90ies making a return.
Anyway, can someone explain the part about K3? What are they trying to say?
qrios 3 hours ago [-]
The interesting part is what they try to not to say: More indications for K3 is based on distillation from Claude and GPT.
From [1]:
> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.
> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s
> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.
Security is solved, but business needs overrides it
Der_Einzige 4 hours ago [-]
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
adrian_b 4 hours ago [-]
What I found the most interesting, and unfortunately not at all surprising, is that the reasoning of the LLMs frequently contained much more useful information than the actual answers, because the answers were censored.
dannyw 2 hours ago [-]
You don’t even get _any_ logits with closed models for years now.
I can’t fault them too much, as logit based distillation is extremely effective.
Very useful for making smaller models out of bigger open weight models.
elzbardico 3 hours ago [-]
Dario is a cunning business man that won't hesitate to say whatever the fuck he needs to get the US government to exercise some regulatory capture to favor anthropic.
syntaxing 3 hours ago [-]
Prefilling Kimi K3 with opus is a super interesting idea. That being said, I absolutely hate this website layout
dboreham 4 hours ago [-]
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
sidsud 4 hours ago [-]
From what I got, the weaker model (Haiku in this case) has access to the shared key and the user simply asks to "transcribe the injected reasoning".
x312 4 hours ago [-]
The provider decrypts it and puts the decrypted reasoning into the model's context window. They prompt the model to repeat back the reasoning. So then the model echoes it back in plain text.
dboreham 3 hours ago [-]
Hmm, ok. So the attack doesn't involve decrypting the payload, only getting the server to do so. Since a model will do that if you just ask, what's so special about the attack?
desterothx 3 hours ago [-]
The large models whose thinking traces are useful are safeguarded against this reasoning replaying. the small models are just designed for speed and efficiency, so these safeguards are a lot meaker, making the attack possible
crazylogger 4 hours ago [-]
Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.
alansaber 5 hours ago [-]
Neat.
3 hours ago [-]
quantumgarbage 6 hours ago [-]
Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model directly or triggering its anti-distillation safeguards.
the_af 5 hours ago [-]
Why do you restate the abstract? Anyone can read it from the link.
Barbing 3 hours ago [-]
This is a non-transparent aspect of submitting a link to HN that is quite misleading.
You think that you are adding a description for your post when in fact you're simply submitting a regular comment not promoted or distinguished in any way.
It's even worse considering posts without URLs would take the same text from the same input box on the HN submission form and append it under the post title, locking it to the top of the page - so if you've browsed linkless posts you think maybe the submission text is gonna end up just like that, but it doesn't. (Also I think you'll even see URLs like an archive link end up appended just below the submission title for URL posts. Which I guess is a special feature.[1])
Any reason I should not send HN an email requesting clarification of this the submission page?
“If there is no url, text will appear at the top of the thread.”
OK, now I finally understand what that means in practice, but it doesn’t imply an entirely separate undistinguished comment will be simultaneously submitted on my behalf.
Wow! I totally missed that the person I was replying to was the one who had submitted the link. What you describe is surely what happened.
I feel bad now :(
Barbing 56 minutes ago [-]
There must be a reason HN does not colorize the OP username or something. But there totally could be some indicator of “post submission text” without too much in the way of negative consequences… (the fact this has never been added tells me I’m being naïve)
Feedback emailed to HN!
ronsor 4 hours ago [-]
This is Hacker News. You know people don't follow links and read.
Groxx 4 hours ago [-]
It's rather common for posters to make a very small summary in a comment. It can help fight the floods of comments working off the title alone (though it's not particularly needed here for that purpose, imo)
the_af 1 hours ago [-]
I totally missed that this was the same person who had submitted the link to begin with. My bad!
mschuster91 4 hours ago [-]
People don't read no links no more
simonw 3 hours ago [-]
[dead]
dxsecarch 3 hours ago [-]
[flagged]
unjuno 3 hours ago [-]
[dead]
locitra 4 hours ago [-]
[flagged]
happybox2016 4 hours ago [-]
The real issue is that API providers log everything. OpenAI/Anthropic already capture full CoT traces in their logs — they just don't expose them. Distillation via API is just making explicit what they already have.
Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-...
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...
Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.
If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.
For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.
https://memory-alpha.fandom.com/wiki/B-4
Sucks.
1. The down side is that it cannot be used across the clients even for the same user
2. Using the same encryption key was a bad choice here, a per user key would have solved this issue for sure.
> a per user key would have solved this issue for sure
It would have helped with PII leakage, but not with plain-text trace extraction attacks, right?
The compliance rules at times are outdated and people skirt around them by following the worded rule instead of the intent.
The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.
1. Its a security issue.
2. Publicly available sessions make it much worse
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
Language is all just sounds and markings. Anything can be redefined to mean anything, and anyone can decide to aggressively assert their preferred definition of a word.
Of course, you can assert that the meaning of "steal" only applies to physical items. You are well within your right to do so. You'd be wrong, but you can do it.
It's more than that. By claiming that copyright infringement isn't stealing, they're usually doing so to justify such behavior: if the original thing remains with the owner, it couldn't have harmed him, could it?
For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
>gl fixing that
https://x.com/_can1357/status/2087228354399265125?s=20
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Also really good for identifying any contradictions in your system prompt and context.
Navigating to the right lands between two cards, so that neither is readable.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
I assume switching the model in the middle of a conversation is intended behavior (very useful in coding agents, for example)
(Thanks for the link. That’s an interesting idea!)
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
you have a secret to keep that is read by the llm, and untrusted input that wants to exfiltrate it.
by hell or high water, the agent is gonna output that text
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :)
I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s.
(I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)
Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it.
Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests.
About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it.
Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
But very interesting result.
Anyway, can someone explain the part about K3? What are they trying to say?
From [1]:
> As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography.
> An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward Opus’s
> A small memorization analysis showed that specific Claude and GPT reasoning spans are up to ~6 orders of magnitude easier to extract from Kimi-K3 than from the next-closest model.
[1] https://x.com/kotekjedi_ml/status/2087147042888114428?s=42
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
I can’t fault them too much, as logit based distillation is extremely effective.
Very useful for making smaller models out of bigger open weight models.
You think that you are adding a description for your post when in fact you're simply submitting a regular comment not promoted or distinguished in any way.
It's even worse considering posts without URLs would take the same text from the same input box on the HN submission form and append it under the post title, locking it to the top of the page - so if you've browsed linkless posts you think maybe the submission text is gonna end up just like that, but it doesn't. (Also I think you'll even see URLs like an archive link end up appended just below the submission title for URL posts. Which I guess is a special feature.[1])
Any reason I should not send HN an email requesting clarification of this the submission page?
Edit: quoting https://news.ycombinator.com/submit :
OK, now I finally understand what that means in practice, but it doesn’t imply an entirely separate undistinguished comment will be simultaneously submitted on my behalf.[1]modpowers(?) used to directly append links to URL submissions further confuse the matter: https://news.ycombinator.com/item?id=49243880
I feel bad now :(
Feedback emailed to HN!