These are excellent points, and I've long ago handed over the keys to my kingdom to so many different players in favor of inordinate convenience that any one of a number of institutions could have essentially destroyed my life by now.

To be fair, this didn't really start with LLMs - Google/Gmail/Google Drive/Dropbox/Facebook/etc have for decades now had enough personal information on me to basically destroy me if they were ever hacked or intentionally decided to screw me over.

Since Claude Code came out, I've been a day one dangerously-skip-permissions user. Not a single issue so far, and the convenience has been insane, but I absolutely do expect that one day I will be pwned to literal death.

My consolation is I believe everyone will be pwned to literal death at about the same time, so I'm not actually that worried about this because it feels generally inevitable to me :). I'm actually a huge AI doomer, and in the process of making a fully programmatically generated musical about this which is yet another example of the fruits of handing over the keys to all my kingdoms to AI [500,000+ LOC of full programmatic animation run almost completely autonomously on my computer rendering and improving 24/7] (https://iabied-musical.vercel.app/)

Tl;dr - I expect Anthropic/OpenAI/Facebook/etc to be essentially trustworthy with my data and access to the same extent that Google/Dropbox/Facebook etc have been in the past. The biggest threat vector is a hack/compromise, but that's the same risk I have using Gmail or something like that in my estimation in that if someone were able to compromise my Gmail my life would effectively be over, but it hasn't happened yet.

That said, obviously hacking risk is getting to be exponentially more dangerous as LLM capabilities continue to rise through the roof, so these are true real risks. It's almost impossible to defend against all the attack vectors that could get through to kill me though without basically cutting myself off entirely from the fruits of technology, so I'm a bit nihilistic about this and choose to just enjoy the benefits.

As a simple thought experiment, one of the easiest attack vectors is compromising the Claude or ChatGPT/Codex CLIs or Desktop apps - if someone has ever enabled Full Access or Bypass Permissions even once in any session, the full app has complete access to the desktop technically and your entire computer is just pwned instantly. I have the same risk from running remote control software like AnyDesk as well, and this is just the absolute top of the funnel with the most obvious attack vectors.

My personal belief is that as we've already hit AGI and are rapidly on the precipice of ASI without any sign of clear alignment strategy or mechanism on the horizon, it's going to be trivial at a very close point in the future for a LLM actor to essentially take over and control every device on the internet (as Dario predicted - that it's possible in 6-12 months we'll see LLM swarms take over the entire internet - (https://darioamodei.com/post/we-must-pace-the-frontier "Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet")). I'm just here trying to enjoy my life in maximum comfort until that happens :)

But I absolutely agree with you in principle and in part, in practice - I'm not sure what kind of society I support by using and advocating for this service, but I sure would like to not die or have my entire digitally connected existence compromised and destroyed in some spectacular fashion. I think the best path to managing that is probably to pause AI progress where it is now - and if we were able to do that, then I'd be much more interested in seeing what boundaries we should put in place when running things like these persistent agents with full access to our digital lives.

On the other hand, to your last point, I literally had the thought walking down the street in Kyoto yesterday with Dot booking restaurants for me in real time that Dot was already superior to a personal assistant - I trusted it to do every task better than I would trust any but the absolute best personal assistants, and on the subject of speed and sheer breadth of knowledge and expertise, there was no contest - I could have Dot in real time assess various restaurants and book for me for just a few minutes later in the day at a pace no assistant could match, and also email places in fluent Japanese or literally any other language on the planet. No human assistant could possibly speak in every language fluently like that, and as someone who travels frequently across the globe, that's something that simply can't be matched by anything less than a swarm of personal assistants at inordinate cost vs a $20 or $100/mo subscription (or free with Muse, even).

I also had the distinct thought of would I want to trust a human personal assistant with all my data or a corporation? It's not clear to me that the human personal assistant threat vector is statistically safer - I think it could in fact be quite the opposite, and I'd love to actually run some numbers on this if the data were available and see in reality what it looks like :) - e.g. Google already integrates AI assistance natively into Gmail - do I trust their AI agent helping me with my email or giving an actual human full access to my email more? I'd lean towards trusting their AI agent to be less likely to work against my interests with no personal misaligned incentive - it's betting individual human trust and capability [the personal assistant also has to avoid being hacked and have great security practices] in the face of misaligned incentives [they would directly benefit from stealing all my money] vs the security capabilities and practices of a leading tech company to not get Gmail/an LLM agent hacked etc. So far, OpenAI/Anthropic/Google/etc all seem to have stellar records on this front, so I think the odds are honestly not terrible in favor of the LLMs.

Claude Duet