'Completely unprecedented' outage causes havoc with IT systems across globe — as it happened
www.abc.net.au
A massive IT systems outage caused by issues with CrowdStrike software affects banks, airports, supermarkets and media companies across Australia and around the world.

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

  • rozodru@lemmy.caEnglish
    422·
    2 months ago

    It’s just amatuer hour across the board. Were they testing in production? no code review or even a peer review? they roll out for a Friday? It’s like basic level start up company “here’s what not to do” type shit that a junior dev fresh out of university would know. It’s like “explain to the project manager with crayons why you shouldn’t do this” type of shit.

    It just boggles my mind that if you’re rolling out an update to production that there was clearly no testing. There was no review of code cause experts are saying it was the result of poorly written code.

    Regardless if you’re low level security then apparently you can just boot into safe and rename the crowdstrike folder and that should fix it. higher level not so much cause you’re likely on bitlocker which…yeah don’t get me started no that bullshit.

    regardless I called out of work today. no point. it’s friday, generally nothing gets done on fridays (cause we know better) and especially today nothing is going to get done.

    • Revan343@lemmy.caEnglish
      121·
      2 months ago

      explain to the project manager with crayons why you shouldn’t do this

      Can’t; the project manager ate all the crayons

    • candybrie@lemmy.worldEnglish
      3·
      2 months ago

      Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.

      • Lightor@lemmy.worldEnglish
        21·
        2 months ago

        Most companies, mine included, try to roll out updates during the middle or start of a week. That way if there are issues the full team is available to address them.

      • rozodru@lemmy.caEnglish
        5·
        2 months ago

        Because if you roll out something to production on a friday whose there to fix it on the Saturday and Sunday if it breaks? Friday is the WORST day of the week to roll anything out. you roll out on Tuesday or Wednesday that way if something breaks you got people around to jump in and fix it.

    • corsicanguppy@lemmy.caEnglish
      1·
      2 months ago

      rolling out an update to production that there was clearly no testing

      Or someone selected “env2” instead of “env1” (#cattleNotPets names) and tested in prod by mistake.

      Look, it’s a gaffe and someone’s fired. But it doesn’t mean fuck ups are endemic.

      • catloaf@lemm.eeEnglish
        3·
        2 months ago

        I’m not sure what you’d expect to be able to do in a safe mode with no disk access.