Replying to everyone to hack HackerNews' Gish Gallop feature (nulla poena sine lege!):

  Alignment is meaningless; as you've noticed, humans aren't all that "morally aligned".

Moral relativism is an attractive proposition when you first examine the topic, but it quickly falls apart; there's a reason it's not even a coherent camp in contemporary philosophy beyond some vagueities from radical post-modernists. Just to go over some of the greatest hits:

- Is what [DICTATOR/MURDERER/CRIMINAL] bad, or merely not to your taste? If the latter, then you have no coherent reason to argue they should be punished. We would never imprison people who don't like vanilla ice cream because 51% of the population does like it.

- If another culture had a deeply held belief to [HORRIBLE_THING] to, say, children, would you just shrug and say "different strokes for different folks"? What if [MURDERER] just had a different culture?

- No, the fact that nature is red in tooth and claw does not disprove morality; we are very, very, very far from our pre-rational, animalistic roots, and to go back now would be unthinkable.

  If the tool needs safety measures it should be kept in a safe enclosure like we do with CNC machines, furnaces, and so on.

Thousands of scientists have been studying this problem for 76 years now, going on 77; your hunch about physical machines does not overrule their findings about the capabilities and tendencies of minds wrought from sand.

  You didn't explain why it's illegal or why distillation is bad.

I think this is just blatantly false, likely based in a misunderstanding of criminal law vs. civil law. Civil courts still deal with legality.

The broader discussion of why distillation is bad and dangerous and immoral is left as an exercise for the reader, as it was above with the parenthetical. It's not a complex argument; I guarantee you understand it if you're reading this.

  Nulla poena sine lege?

The same thing as above -- the fact that laypeople can not think of a criminal charge that they've heard on Law & Order that corresponds to this behavior does not mean that it's legal. It's textbook fraud, regardless of what particular detail you focus on.

  I'm always wondering when "distillation" comes up how feasible it is, or if it's just BS... Or am I missing something here that makes real "distillation" feasible?

I think the fact that it's happening at such a large scale is proof that very smart, well-resourced labs in China (the producers of the world's best OS models, including the incredible GLM-5.3-Flash) think it's feasible. I'm not sure it's productive to question them in the absense of any indication to the contrary.

This is a great question still, not trying to shut you down. But I think the fundamental issue is a misunderstanding of what distillation is -- it's not directly stealing literal atomic parameters and piling them up. They might try to focus on substructures within these massive networks, but even that isn't strictly necessary for a distillation attack.

  Source for 1? Are we sure those aren't hallucinations?

Sorry, I never linked it! This is from the latest Anthropic safety report (of "Anthropic Houtis build missile" fame), and no, these cannot be hallucinated -- the leaked secrets were inputs, not ouputs. https://www.anthropic.com/threat-intelligence-report-septemb...

  Like Anthropic and OpenAI are? After all, didn't they distill all the information in the world into their model(s)?

This is just blatant word games, sorry. I'm sure intended in good faith, and I understand the impulse -- I consider myself a radical anti-IP slacktivist, after all. But "both things involve information transfer" is just not a coherent point; lots of things fit that description.

  I mean, if they get to distill other's IP, why can't others distill their IP?

These are cyberattacks. Yes, anyone can cyberattack cyberattackers. But, y'know... an eye for an eye...

  Yes, wont somebody please think of the shareholders whose IP had been stolen...

I am not at all concerned with the value of the resulting artifacts as assessed by the (already totally unhinged) NYSE et. al. I am concerned about user respect, law following, truth telling, blatant cyber warfare at a time of rising tensions, accidental data leakages at a scale that'd be hard to fathom 5 years ago, bad-faith public postures, and a general distaste for fraud.

How GLM built its own inference infrastructure