we suffered some extremely unexpected downtime while I deployed a trivial change (a reverse proxy from http://awful.systems/archives to http://these.awful.systems/archives) to prod

the downtime was unrelated to the deployment change; instead, it seems like lemmy-ui started crashing because it couldn’t render the app icons it uses when saved as a home screen app on mobile. it uses a fairly heavy dependency to do this, and has no error handling in case the source icon data is corrupt, which causes it to crash on every request (resulting in a 503 Service Unavailable error for everyone who tried to access awful.systems during this outage)

since I don’t know how that corruption occurred or why it was persistent (the app icon data should be fully static as part of the Nix store as far as I know), so until I can dig in I’ve disabled generating app icons for our instance. since it seems like we’re the first ones to hit this bug, I’ll do my best to keep the patch upstreamable so other lemmy instances can benefit from the fix

  • David GerardA
    link
    English
    41 year ago

    if anyone tries to make you maintain a patch set, be sure to call it Awful

    • @selfOPMA
      link
      English
      31 year ago

      awful.patch, a diff file that quickly grows to be larger than the base lemmy code

      • @kuna
        link
        English
        3
        edit-2
        1 year ago

        deleted by creator

        • @selfOPMA
          link
          English
          51 year ago

          I’ve still got dibs on LessJank