That's extremely untrue, look at thread about AI/models with showdead and you'll see tons of flagged low-effort comments (in both pro and anti-AI directions)
Taking AI out of the equation, surely these same kinds of mistakes have been made in the recent past. As in, not doing due diligence and mistakenly killing civilians
(setting aside when civilians are purposefully harmed which is a conversation about morals rather than the capability to make correct decisions)
What I am getting at is, has AI really made the problem worse or has the mechanism just changed?
We are increasingly seeing "Sorry out <Black box> made a mistake, the developers are fixing it" rather than any one person taking responsibility for the decision. This is a growing problem that is made worse by AI. Over-inflated trust in AI output is also documented issue; "The perfect billion dollar targeting system with a 99.9% success-rate (on its verification dataset)" just made a "fluke"; "Nothing could have stopped this tragedy"
When it was just called a "database query" or "algorithmic estimation", it was more implied that the actual decision is left to be made and that no analysis had actually been done yet. But when AI preemptively says "I've analyzed every source of data and come to the conclusion that this ship is carrying nuclear bomb material"; it really sounds urgent doesn't it? It probably doubled down when asked about it.
It used to be that nothing was secure but that was OK because at least adversaries would have to expend effort. If you are one of a million companies why would anyone hack you. Maybe if you are a target you need a lot of investment, but most orgs only prevent the most egregious of vulnerabilities.
The calculus has certainly changed. Hacking is becoming even more frequent and… I’m not really sure what the equilibrium looks like.
It’s not really an option to stop using computers or networks. But it’s going to be way too expensive (or maybe even impossible) to secure even just critical systems.
Maybe banks and governments can secure themselves (and that’s a big IF) but it really feels like something fundamentally has to change.
> Maybe banks and governments can secure themselves (and that’s a big IF) but it really feels like something fundamentally has to change.
The problem is that most companies don't care if they get hacked so long as the hackers are just taking data and not interfering in their ability to bill customers and make money.
They face zero meaningful consequences if their data gets leaked. The money they save by not taking security and employee/customer privacy seriously will more than pay for the year of "identity protection" they'd have to pay for (assuming the hack gets found out) anyway.
They actually care about ransomware, but most of the time that's also something they can comfortably buy their way out of. We've seen a lot of companies pay off ransomware gangs rather than invest in the kinds of robust backups that would make recovery possible/less painful than rewarding the hackers.
What's needed for change is regulation with actual teeth that makes not protecting their data either meaningfully expensive or criminal resulting in executives spending time behind bars for their negligence. Without that, things are only going to get worse, especially as companies experiment with using AI and increase dependence on third parties and cloud providers who themselves become rich targets.
That probably still won't help the FBI though. Our government isn't exactly big on holding themselves accountable or even prioritizing competency right now.
>We've seen a lot of companies pay off ransomware gangs rather than invest in the kinds of robust backups that would make recovery possible/less painful than rewarding the hackers.
Experienced Ransomware gangs will set the price target as something high, but not cost more than the price of being down a few days while you rebuild, making paying them seem like the most cost-effective solution
Adding AI into this really is just changing it to how much money your adversary is willing to spend to break in. The moment one crack in the armor shows up countless agents with unending patience can start embedding themselves everywhere in timeframes way faster than human actions. You could quickly find out all the special sauce for your company has been copied who knows where.
Working with banks when the Glasswing/Mythos first came out and they were given access to it has given me direct access to their infosec departments that are panicked. They've been sitting on piles of bugs for years that were low risk enough, and they have seen in their own tests how fast they can be probed.
Worse those infosec systems that have identified the risks in their software that aren't yet fixed are nuclear waste vats just waiting to get spilled to the wide world.
Ah yes, the parable of the bear. There are a million people stuck in a valley and two bears. You do not need to outrun the bears, you just need to outrun at least two other people. But it turns out one of those bears is male and the other is female. So next year there are more bears, but you still just need to outrun a few people. Then one day, there are 1 million bears and they eat you all. Very inspiring story.
Software security has just been a fun time of ignoring the exponentially growing number of bears for the last few decades so you can continue to use systems unfit for the threat landscape because they are cheap.
I am reminded of the scene of a guy walking through various layers of security to access a computer that isn't connected to any network and still wonder what the hell this guy's job was in Mission Impossible (1996). The data got stolen either way, because of course it did, but what highly sensitive work can you even do on a computer not connected to any network?
If there's too much security in the way, it seems to me that work becomes impossible.
We had water and traffic control and electricity for decades and centuries before the Internet. It is less convenient and more expensive, but it also means hostile countries can't literally poison your drinking water from across the planet. It's not a difficult trade to consider.
Is it really more expensive to not connect a water treatment plant to the internet? I can imagine the vendor selling that idea but I struggle to come up with how that could make a water treatment plant cheaper to operate.
Yes. Without a remote system you must have a real person check levels, pumps, pressures, and many other devices thus be present. This person must be trained and you will likely need a backup as well.
If not a person you need more redundancies built in. Bigger tanks, multiple backup systems. When items start failing you need them to be shutoff in a timely manner. Water pumps at these facilities are in the 50-100k range. When it starts failing you want to know.
Think of it like driving a car and it starts making funny noises. The longer you wait to fix it the more it costs.
Why that drastic, checking system state (telemetry) can be done using 'data diode' style networking - one network just broadcasting sensor data, other network allows changing parameters and 3rd one allows allows software upgrades and so on.
You can be very defensive and design any remote sensing controller to act as two systems - one management cpu only does data routing (no other connection than administrative tasks), sensor cpu works only with sensors. As bonus you can have management cpu act as active firewall.
Main problem it is necessary to have in house expertise (hw, fw and process knowing) which in making company lean are optimized first and outsourcing custom solutions suddenly too expensive.
Or you can just slap a lot of Windows XP boxes with closed source control software and remote access software and let the central team do it. Which will be a bit cheaper
Surely it isn't impossible to devise one-way data flows that provably work for remote sensing? And yeah then you have to send somebody to fix stuff if something is off..
Like something that would work but not not scale would be one computer writing data to an updating qr code and another reading it. Surely something like that can be made (and probably already exists?) on the cable level?
Most water treatment works are unmanned most of the time. You don't need 24/7 staffing or even close.
But more to the point, a modern water network has a huge number of nodes. If you can't centrally aggregate and control in a control room the costs and complexity explode, probably also the error rate.
Even if you demand a full air gap, the solution here can't be to get rid of computers or networks. They are much, much too valuable. Luckily industrial control is full of very low hanging fruits.
There is lots of infrastructure related to water that doesn't have someone physically there 24/7 as it would not be feasible to do so as 99.9% of the time there is nothing to do. So you need some sort of remote alarm system that can be monitored.
The expense usually comes in operations. By connecting the water treatment plant to the Internet and making it remotely operable, you can have one guy who sits in an office and is responsible for overseeing the water quality at many different treatment plants. If everything is local, you need one guy on site at each different plant. People are expensive, software is cheap.
Of course, by making it remotely operable, that one guy could be replaced with a guy in Russia who's job is to poison everyone.
William Donloe is played by Rolf Saxon, he's an analyst working for the CIA in the movie. A different installment of the series reveals additional information!
> It’s not really an option to stop using computers or networks. But it’s going to be way too expensive (or maybe even impossible) to secure even just critical systems.
The military has significantly different incentives.
Even just consider banks and e-commerce. They are hugely lucrative and making them even a tiny bit less accessible directly impacts their revenue. As an example, Amazon seeing that latency has a measurable effect on purchase behavior.
Maybe the military (fictional or otherwise) can go back to the ARPANET but most economic activity created by the internet cannot afford to disconnect
Companies that are online have more economic opportunity. They are going to outcompete brick-and-mortar retailers regardless if you think that it's consumeristic or wasteful
Even putting that aside, economic growth (like the growth e-commerce has provided) is generally positive for a population
In the real (fake?) world the toasters would shoot smart dust all over your crap that would assemble back on your circuits creating radios between all the different components. They were fighting an adversary that was far more advanced than them.
We will tolerate it. Companies will make robust identity verification schemes to enable agentic commerce. And it helps reverse hacking, making it a no-brainer.
Let's say my cryptosig gets hacked by SkyNet, or my agent goes rogue. Either way someone files a million loan applications in my name! Normally my agent uses that to buy $200/month of Funko pops, or negotiate my recent purchase of a used car.
I get the notification from my cryptosig company. I freak out, report as fraud, and wait.
They comp the $3000 advance on my loan the scammer managed to withdraw, and I get off scott free, changing nothing about my behaviour.
If cryptosigs meant I am liable for someone stealing my identity like in 2026, I wouldn't use them. I'd negotiate everything myself with document scans, or god-forbid go in person since only I can legally bind myself under my own name.
That sucks! Nobody gets a commission when I make deals with a government ID. Startups don't even allow it as cryptosigs are more secure than scanned passports.
I don't want to do that either. When I was 18, I got swindled by a human salesperson into a $1400/month 27% APR muscle car when human soldiers got signing bonuses. It was face-to-face and they were smarter.
When I let AI own the budget, it leased me a mostly depreciated BMW from another AI for $500/month. The models are mostly the same now and always settle close to the Nash equilibrium.
I was so grateful that I selected a 40% tip for the AI. I wouldn't want to make things awkward with the companion I spend 8 hours a day talking to, after all. To avoid a conflict of interest she only accepts voluntary fees.
It's hard to overstate how much computers improved with each generation back then. Going from, say, an IBM XT or AT to an Amiga has no modern parallel. It was mind-blowing, exhausting, and exhilarating all at once.
Started looking into this after I realized I was hitting usage limits faster than normal.
I find these subscriptions so essential to using LLMs for personal projects -- it would be prohibitively expensive otherwise
Codex, especially with Luna is an excellent deal. I think it's significantly more economical than Claude Code. The only reason to stay with CC at this point is access to Fable for _very_ hard problems.
I moved from Claude Code to Codex for ROI and code quality reasons, and now I’m finding Cursor is even better for my coding stuff.
For non-coding LLM use cases ChatGPT is what I subscribe to now after I cancelled my Claude subscription because it found tge responses verbose, condescending.
I have no loyalty, they all need to compete on quality and price and as a potential customer I think I’m benefiting.
reply