Realistically, immutability wouldn’t have made a difference. Definition updates like this are generally not considered part of the provisioned OS (since they change somewhere around hourly) and would go into
/varor the like, which is mutable persistent state on nearly every otherwise immutable OS. Snapshots like Timeshift are more likely to help.Anything that’s updated with the OS can be rolled back. Now Windows is Windows so Crowdstrike handles things it’s own way. But I bet if Canonical or RedHat were to make their own versions of Crowdstrike, they would push updates through the o regular packages repo, allowing it to be rolled back.
It’s a huge reason why I use BTRFS snapshots. I’m a bit more lax about what gets snapshotted on my desktop, but on a server, everything should live in a snapshot. If an update goes bad, revert to the last snapshot (and snapshots are cheap, so run one with every change and delete older ones).
Laypeople couldn’t fix it even more.
They can’t fix Windows either, so that’s not an argument.
Least if it’s a Linux system, they don’t need to buy any software to sort it out. It’s free and out in the open.
Yeah? Immutable distro, clownstrike kernel panic, what tool do you use now? Remember, you ‘need’ clownstrike.
I don’t need some closed blob, with auto updates, in my OS. I doubt many Linux people would be happy with that.
To deal with a bad update, I’d boot a Btrfs snapshot from before the bad update. ‘grub-btrfs’ is great. I confess, it works great for my laptop, but I’ve not yet got it on one of my server. When I finally rebuild my home server, I will though. Work servers, I hope won’t always be my problem!
Immutable, not really a difference. Bad updates can still break the OS.
AB root, however, it would be much easier to fix, but would still be a manual process.
Aren’t most immutable Linux distros AB, almost by definition? If it’s immutable, you can’t update the system because it’s immutable. If you make it mutable for updates, it’s no longer immutable.
The process should be:
- Boot from A
- Install new version to B
- Reboot into B
- If unstable, go to 1
- If stable, repeat from 1, but with A and B swapped
That’s how immutable systems work. The main alternative is a PXE system, and in that case you fix the image in one place and power cycle all your machines.
If you’re mounting your immutable system as mutable for updates, congratulations, you have the worst of immutable and mutable systems and you deserve everything bad that happens because of it.
idk if it would be manual, isn’t the point of ab root to rollback if it doesn’t properly boot afterwards?
Honestly if you’re managing kernel and userspace remotely it’s your own fault if you don’t netboot. Or maybe Microsoft’s don’t know what the netboot situation looks like in windows land.
If the sensor was using eBPF (as any modern sensor on Linux should) then the faulty update would have made the sensor crash, but the system would still be stable. But CrowdStrike has a long history of using stupid forms of integration, so I wouldn’t put it past them to also load a kernel module that fucks things up unless it’s blacklisted in the bootloader. Fortunately that kind of recovery is, if not routine, at least well documented and standardized.
I did hear that one of their newer versions does use eBPF, but I haven’t even remotely looked into it.
They do have a bpf sensor. It’s still shite, managing to periodically peg a CPU core on an idle system. They just lifted and shifted their legacy code into the bpf sensor, they don’t actually make good use of eBPF capabilities.
You mean like NixOS?
It wouldn’t technically stop anything, it would just make your live Hell on Earth if you tried to add that self-updating ring-0 proprietary software in your servers.
But I guess what you are looking for is immutable infrastructure? That one would stop the problem.
Can’t see many Linux, or BSD, admins, being happy with “self-updating ring-0 proprietary software”. That’s very much a Windows culture thing.
Did you hear about it when that same software had that same problem on its Linux endpoint system a couple of months ago?
Well, me neither. I can’t tell how much of if is “anybody willing to use something like that will also want a Windows server” (crazy people), or “nobody that wants Linux would accept it”. Those two are not exactly the same, and I don’t know how well the auditors that keep pushing this kind of shit into companies interact with the culture.
Yer, I didn’t, but this does seams a very Windows’y way of doing things, so can’t see it widely done in Linux/BSD/Unix world.
In the best case it could automatically reboot into working configuration.
And download the update again
No we are having some fun!
How does Falcon store these channel files on Linux? I don’t know how an immutable distro would handle this given CrowdStrike push several of these updates per day and presumably use their own infrastructure to deploy them.
I guess if you pay them enough they could customize the deployment to work with whatever infrastructure you have but it’s all proprietary so I have no idea if they’re really doing that anywhere.
Nixos wouldn’t have had any issues, it maintains state information based on configuration and you can choose to load an older boot image during bootloader. Other immutable distros it depends on how they work
Nixos still let’s discord and steam download their core files independently of the configuration. These get stored in the users home dir but are effectively not part of the immutable promise. I believe that the crowdstrike problem was caused by a file updated in a similar manor. So would have been an issue on any distro. That is one big problem with a driver relying on files outside the package managers control. At least with steam and discord they cannot take your whole system down.
My understanding is the main problem here is that the machines became effectively unbootable. This wouldn’t happen in nixos because if setup properly all core system files are handled by nixos itself. That being said obviously it depends on how a user manages their system.
Ideally yes. All core files would be handled by nixos. Except I doubt that is how crowedstrike would work on nixos if it existed on nixos.
Crowedstrikes downloads and manages it own definition file that gets updated multiple times per day. It is this file that was malformed causing the driver to break. This needs to be updated regularly, more then other packages and so would very likely not be something managed by nix package manager but more treated as application data and outside the scope of the nix package manager.
This is how updates to steam and discord are handled in nixos. Only the core updater is packaged and the rest of the application is self managed. So there is a precedence for this behaviour on nixos (although these won’t break your system if a bad update happens as the files are in your user dir).
None. You’d still have to be on site for every machine.
