A week of downtime and all the servers were recovered only because the customer had a proper disaster recovery protocol and held backups somewhere else, otherwise Google deleted the backups too

Google cloud ceo says “it won’t happen anymore”, it’s insane that there’s the possibility of “instant delete everything”

  • Simon@lemmy.dbzer0.com
    link
    fedilink
    arrow-up
    2
    ·
    1 year ago

    Just an FYI in case you don’t follow Cloud news but Google has deleted customers accounts on multiple occasions and has been for literal years. This time they just did it to someone large enough to make the news. I work in SRE and no longer recommend GCP to anyone.

    • Hirom@beehaw.org
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      They had backups at multiple locations, and lost data at multiple (Google Cloud) locations because of the account deletion.

      They restored from backups stored at another provider. It may have been more devastating if they relied exclusively on google for backups. So having an “offsite backup” isn’t enough in some cases, that offsite location need to be at a different provider.

      • Tangentism@lemmy.ml
        link
        fedilink
        arrow-up
        0
        ·
        1 year ago

        It may have been more devastating if they relied exclusively on google for backups.

        Which is why having any data, despite the number of backups, on a cloud provider shouldn’t be seen as off-site.

        Only when it is truly outside their ecosphere and cannot be touched by them should it be viewed as such.

        If that company didn’t have such resilience built into their backup plan, they would be toast with a derisory amount of compensation from Google.

        • Hirom@beehaw.org
          link
          fedilink
          arrow-up
          0
          ·
          1 year ago

          Having a backup at a cloud provider is fine, as long as there is at least one other backup that isn’t with this provider.

          Cloud provider seems to do a good job protecting against hardware failure, but can do poorly with arbitrary account bans, and sometimes have mishaps due to configuration problems.

          Whereas a DIY backup solution is often more subject to hardware problems (disk failure, fire, flooding, theft, …), but there’s no risk of account problem.

          A mix is fine to protect against different kind of issues.

          • Tangentism@lemmy.ml
            link
            fedilink
            arrow-up
            0
            ·
            edit-2
            1 year ago

            as long as there is at least one other backup that isn’t with this provider.

            Which is exactly what I was saying.

            Any services used with a cloud provider should be treated as 1 entity, no matter how many geo-locations they claim your data is backed up to because they are a single point from which all those can be deleted.

            When I was last involved in a companies backups, we had a fire safe in the basement, we had an off-site location with another fire safe & third copies would go off to another company that provided a backup storage solution so for all backups to be deleted, someone had to go right out of their way to do so. Not just a simple deletion of our account & all backups are wiped.

            That company had the foresight to do something similar & it’s saved them. [edited - was on the tube when I wrote this and didnt see the autocorrect had put ‘comment’, not ‘company’]

    • Chozo@fedia.io
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      Money. It’s a lot cheaper to let somebody else maintain your systems than to pay somebody to create and maintain your own, directly.

          • GolfNovemberUniform@lemmy.ml
            link
            fedilink
            arrow-up
            0
            ·
            1 year ago

            Google is one of the most privacy invasive companies in the world. And judging by encryption standards, terms of service and privacy policies

            • settoloki@lemmy.one
              link
              fedilink
              arrow-up
              1
              ·
              1 year ago

              Are you sure you’ve not just read bad stuff without verification on the internet and feel the need to chime in on something you don’t fully understand?

                • settoloki@lemmy.one
                  link
                  fedilink
                  arrow-up
                  1
                  ·
                  1 year ago

                  Me too as a programmer that uses Google cloud to store government information. Which bit of the policy says they are going to access your data, shouldn’t take you long to link it to me if you read them as much as you say. Unless what you’re actually doing is spreading misinformation and bullshit.

            • Pup Biru@aussie.zone
              link
              fedilink
              arrow-up
              1
              ·
              1 year ago

              and you know the security standards that are achievable on google cloud entirely negate your point right? their cloud offering is a totally different beast

  • Mossy Feathers (She/Her)@pawb.social
    link
    fedilink
    arrow-up
    0
    ·
    edit-2
    1 year ago

    They said the outage was caused by a misconfiguration that resulted in UniSuper’s cloud account being deleted, something that had never happened to Google Cloud before.

    Bullshit. I’ve heard of people having their Google accounts randomly banned or even deleted before. Remember when the Terraria devs cancelled the Stadia port of Terraria because Google randomly banned their account and then took weeks to acknowledge it? The only reason why Google responded so quickly to this is because the super fund manages over $100b and could sue the absolute fuck out of Google.