Is the site down, or is it me? Introducing new FSF Uptime Kuma instance — Free Software Foundation — Working together for free software
Is the site down, or is it me? Introducing new FSF Uptime Kuma instance
Over the last couple of years, waves of botnets have occasionally taken down our sites. This is not a problem unique to us, but one that the Internet has been facing as a collective. The origins of the attacks range from aggressive LLM scrapers, vulnerability scanners, poorly optimized CI/CD servers, and beyond. We have made significant progress fighting these attacks, but there is always more work to be done to improve our systems and strengthen our defenses. In our efforts to combat the botnets, we optimized several detection rules to ban abusive behavior. We found the upper limit of fail2ban and replaced it with reaction, an efficient alternative with our configuration that uses ipset. We also split several monolithic machines into many separate machines so that when a web service is overwhelmed the other functions of the service do not go down with it.
We found quite a few ways to respond to and prevent botnet attacks, but still faced a significant related challenge: communicating when a website or service is down. For the longest time, our main method for communicating what was going on to the public was through writing articles, posting to our @fsfstatus microblogging account, or responding to IRC messages and email. While these methods are effective, communication with users was dependent on someone being online and having the time to write a response, resources which were often already occupied with the attacks. This lack of capacity to respond often left users wondering, "Is the site down or is it just me?"
Communication was a bit better internally, as we have always had some sort of monitoring system in place, but there was room for improvement. When I joined the FSF, we were using Munin and Nagios, and would receive reports from Ansible, cron jobs, and other various monitoring systems. These solutions got the job done, but they often gave more information than was useful for the average user or not safe to share with non-FSF staff and volunteers. Additionally, the data provided by these tools is often hard to understand if you are not already familiar with how these solutions work. Now, we primarily use Prometheus for an advanced view into our infrastructure, along with Uptime Kuma.
What is Uptime Kuma exactly? Uptime Kuma is a human-readable, automated monitoring addition to our systems. The interface is easy to understand at a glance with green, red, and blue colors for each minute. Red means that a website or service took longer than forty-eight seconds to respond. Green represents a website or service that took less than forty-eight seconds to respond. Blue is for a website or service that is undergoing maintenance. This nearly up-to-the-minute interface is much faster than any of us writing microblog status updates. Uptime Kuma helps anyone gain insight into whether an FSF-hosted site is up or down in reality. You can check out our recently-launched self-hosted Uptime Kuma instance at https://status.fsf.org/. When you see the page, you will also likely say, "Wow! The FSF and GNU sure do run a ton of services!" and you would be right.
If you check Uptime Kuma and see that a website or service you are trying to use is red, what should you do? First, confirm that the service is not flapping (going down and coming back up automatically), which is normal. If a website goes down and does not come back up after ten to thirty minutes, you can check @fsfstatus@hostux.social on the fediverse for periodic updates or check in with #fsfsys on Libera.Chat IRC to get more information. We likely already know about the issue due to our automation, but we are glad to provide updates when we can.
If you maintain websites and services, and are looking for a simple way to communicate publicly with your users, consider using Uptime Kuma or another free software solution instead of choosing a proprietary monitoring solution. If you are looking for further insight, we also recommend Prometheus.
*"Kodiak Bear" © by David Menke. This image is dedicated to the public domain under a Creative Commons CC0 No Rights Reserved license.*