: Fault in DynamoDB system cascaded through AWS services, knocking major sites offline for hours
      • MelodiousFunk@slrpnk.netEnglish
        26·
        2 months ago

        If I had a nickel for every time clearing the ARP tables fixed a problem, I’d have a shitload of nickels.

          • MelodiousFunk@slrpnk.netEnglish
            29·
            2 months ago

            These things happen when a skinflint company contracts out network setup for a decade, gets acquired by another skinflint company who axes the contractors and doesn’t hire on-site network personnel, gradually builds out infra on top of the unsupported foundation, and then hires c suite buddies who want to bring in their own people to further muddy the waters.

            • sleepmode@lemmy.worldEnglish
              3·
              1 month ago

              Like every MSP ever. When your CEO that started the company in college suddenly shows up in a green Lamborghini it is time to spruce up the resume.

  • GreenKnight23@lemmy.worldEnglish
    126·
    2 months ago

    oh sure, when they fuck up DNS it’s a “race condition”.

    when I fuck up DNS it’s a “fireable offense”.

  • falseWhite@lemmy.worldEnglish
    103·
    2 months ago

    That’s what you get when you let go hundreds of employees from your cloud computing unit in favour of AI.

    I hope they end up having to compensate all the billions of losses they caused to all the businesses and people.

      • falseWhite@lemmy.worldEnglish
        37·
        2 months ago

        They do have contracts and are obligated to provide a certain “up time”, which is usually 99% or so. If they fail to provide that, they are liable to compensate for the losses.

        Or do you think that Amazon is above the law and no other company could sue them?

        It all depends on what kind of contracts they have.

        • Onomatopoeia@lemmy.cafeEnglish
          26·
          2 months ago

          Much of this stuff is automatic - I’ve worked with such contracted services where uptime is guaranteed. The contracts dictate the terms and conditions for refunds, we see them on a monthly basis when uptime is missed and it’s not done by a person.

          I imagine many companies have already seen refunds for outage time, and Amazon scrambled to stop the automation around this.

          They’ll have little to stand on in court for something this visible and extensive, and could easily lose their shirt with fines and penalties when a big company sues over breech when they choose to not renew.

          Just cause they’re big doesn’t mean all their clients are small or don’t have legal teams of their own.

        • WASTECH@lemmy.worldEnglish
          11·
          2 months ago

          These contracts do not stipulate reimbursement for lost revenue. The “uptime guarantee” just gets you a partial discount or service refund for the impacted services.

          It is on the customer to architect their environment for high availability (use multiple regions or even multiple hyperscalers, depending on the uptime need).

          Source: I work at an enterprise that is bound by one of these agreements (although not with AWS).

          • CheezyWeezle@lemmy.worldEnglish
            9·
            2 months ago

            SLA contracts can have a plethora of stipulations, including fines and damages for missing SLO. It really depends on how big and important the customer is. For example, you can imagine government contracts probably include hefty fines for causing downtime or data loss, although I am not involved with or familiar with public sector/ government contracts or their terms.

            You can imagine that a customer that is big enough to contract a cloud provider to build new locations and install a bunch of new hardware just for them, would also be big enough to leverage contract terms that include fines and compensation for extended downtime or missing SLO.

            I work at a data center for a major cloud provider, also not AWS

        • BakerBagel@midwest.socialEnglish
          9·
          2 months ago

          Amazon has more money than most countries. They can outlast any company in court, or just ban you from their services in the future.

          • Onomatopoeia@lemmy.cafeEnglish
            11·
            2 months ago

            Depends on who we’re talking about. Companies like finance orgs are all about legal contracts and would be able to hold their feet to the fire.

            You don’t want to go to court against a finance company or any very large org where contract law is their bread and butter (basically any large/multinational corp).

            Amazon’s not hosting just small operations.

        • BCsven@lemmy.caEnglish
          6·
          2 months ago

          Most services have a clause that they are not liable for unforseen issues… Depends how good the lawyers were when formalizing the contracts.

          • Passerby6497@lemmy.worldEnglish
            4·
            2 months ago

            Good luck arguing that a missed config counts as an ‘unforeseen issue’. If they go that route, people will be all over them for not being SOC compliant wrt change control.

            • BCsven@lemmy.caEnglish
              1·
              2 months ago

              They can try to argue that latency issue and the stale state were an unknown / unanticipated problem. Like when half of Canadas Rogers network went down affecting most debit payment systems. Testing of routing showed it OK, realworld flip went haywire.

        • Passerby6497@lemmy.worldEnglish
          4·
          2 months ago

          99% uptime in a year gives you 3.65 days of downtime, which I think would still be within SLA (assuming nothing else happened this year). Though, once you get to 1 9 reliability (99.9%), you’ve got a shift and change you can be down before you breach SLA.

          If their reliability metrics are monthly, 99% gets you less than a shift of down time, so they’d be out of SLA and could probably yell to get money back.

        • 87Six@lemmy.zipEnglish
          3·
          1 month ago

          Oh yea, other companies will sue them, and when amazon completely fails they will be bailed out with consumers’ tax money. Or did we already forget that’s what happens?

    • Possibly linux@lemmy.zipEnglish
      19·
      2 months ago

      Mistakes happen with or without AI

      The problem is that the current internet is structured in a way that creates high risk systems that can cause a massive outage. We went from having thousands of independent companies to a handful of massive ones. A mistake by a single company shouldn’t be able to black out half the internet.

      • amino@lemmy.blahaj.zoneEnglish
        15·
        2 months ago

        Signal is definitely part of the fun internet, they just decided to rely on AWS due to techbro culture I assume?

        • dubyakay@lemmy.caEnglish
          4·
          2 months ago

          They rely on AWS due to favourable contract in hosting it, and also proving the proof of concept that they can be hosted securely on a hostile provider, without the provider having any clues at all in what data is being sent between the parties.

          • amino@lemmy.blahaj.zoneEnglish
            6·
            2 months ago

            sure, proving to the audience that you can kick yourself in the nuts over and over while maintaining the privacy of your testicle’s innards is impressive from a biological standpoint but it still looks stupid to a normal person. I don’t hate signal, I will continue using it but this and their crypto scam makes me doubt some of their choices and how they’ll operate in the future

  • Pokexpert30 🌓@jlai.luEnglish
    37·
    1 month ago

    Just one more layer bro, just one more automated planning system bro and this time it will be entirely faultless please bro one more layer

  • TommySoda@lemmy.worldEnglish
    37·
    2 months ago

    This is purely anecdotal, but I have been running into a lot of DNS issues over the past couple months where I work. 3 of the computers and even one of the laptops for remote work were having DNS issues that needed to be fixed. One even needed Windows reinstalled after fixing the DNS issue (Which was probably unrelated, but worth mentioning)

    I’m honestly starting to think that the internet in general might be imploding. Not sure why, but replacing so many developers and programmers with AI might be responsible. Who knows, but it’s definitely very strange.

    • Possibly linux@lemmy.zipEnglish
      57·
      2 months ago

      The biggest issue is how centralized the internet has become. It went from a bunch of local servers to a handful of cloud providers.

      We need to spread things out again

      • Canopyflyer@lemmy.worldEnglish
        1·
        1 month ago

        But but Bezos has to pay for another rocket and yacht and he just got married!!! Think about his quarterly statement! My god are you heartless!!!

        /s

        (just in case it’s not obvious)

    • ubergeek@lemmy.todayEnglish
      26·
      2 months ago

      A huge problem are developers who lack a fundamental understanding of how the internet even works. I’ve had to explain how short, unqualified names resolve vs how fqdns resolve. Or why even you may not be able to reach another node in your proverbial cluster, because they are on different subnets. Or, why using GUIDs as hostnames is a generally bad idea, and will cause things to fail in unpredictable ways, especially with deeply nested subdomains.

      • GreenKnight23@lemmy.worldEnglish
        17·
        2 months ago

        I have worked with too many devs that didn’t even know what the 7 layers/OSI are or why they exist.

        they didn’t know what a network port was used for and why it’s important to not expose 3306 to the internet.

        they couldn’t understand that fragmentation of a message bus occurs when you don’t dedupe the contents.

        you know, morons.

  • SayCyberOnceMore@feddit.ukEnglish
    14·
    2 months ago

    I’m glad these things happen… it keeps everyone aware that cloud is fragile and Plan B should be considered for mission critical tasks.

    I’m also hoping that it will improve cloud resiliency because a complete / partial restart of cloud systems needs a whole different approach than maintaining a running system.