we suffered some extremely unexpected downtime while I deployed a trivial change (a reverse proxy from http://awful.systems/archives to http://these.awful.systems/archives) to prod

the downtime was unrelated to the deployment change; instead, it seems like lemmy-ui started crashing because it couldn’t render the app icons it uses when saved as a home screen app on mobile. it uses a fairly heavy dependency to do this, and has no error handling in case the source icon data is corrupt, which causes it to crash on every request (resulting in a 503 Service Unavailable error for everyone who tried to access awful.systems during this outage)

since I don’t know how that corruption occurred or why it was persistent (the app icon data should be fully static as part of the Nix store as far as I know), so until I can dig in I’ve disabled generating app icons for our instance. since it seems like we’re the first ones to hit this bug, I’ll do my best to keep the patch upstreamable so other lemmy instances can benefit from the fix

  • @selfOPMA
    link
    English
    31 year ago

    awful.patch, a diff file that quickly grows to be larger than the base lemmy code

    • @kuna
      link
      English
      3
      edit-2
      1 year ago

      deleted by creator

      • @selfOPMA
        link
        English
        51 year ago

        I’ve still got dibs on LessJank