04 July 2025
Article published by: wxie
Date of publication: 04 July 2025
This is an email from info@fsf.org.
The FSF SysOps team consists of two full-time tech team employees and a handful of dedicated volunteers. A large part of our work is running the software and physical servers that host websites and other services for GNU, FSF, and other free software projects, including virtual machines for the browser extension JShelter, the desktop environment and software collection KDE, and Sugar Labs, an organization that creates learning tools for children. We recently counted seventy different services, and have a dozen physical servers across two Boston-area data centers.
Since we last wrote, much has happened, and while I’d love to talk about all of it, including the process of deploying four new servers to our data centers, I want to focus on the huge task of maintaining our services in the face of ongoing (and increasing) distributed denial of service (DDoS) attacks. A DDoS attack typically happens when attackers control thousands or millions of machines and get them all to send requests or other traffic to a target server. Then, the server gets overwhelmed with processing those requests and fails to respond to requests from legitimate users. A common way of defending against a DDoS attack, which we often use, is to figure out a way of identifying which IP addresses are sending requests as part of the DDoS, and then have the server ignore requests from those IP addresses.
Our infrastructure has been under attack since August 2024. Large Language Model (LLM) web crawlers have been a significant source of the attacks, and as for the rest, we don’t expect to ever know what kind of entity is targeting our sites or why.
In the fall Bulletin, we wrote about the August attack on gnu.org. That attack continues, but we have mitigated it. Judging from the pattern and scope, the goal was likely to take the site down and it was not an LLM crawler. We do not know who or what is behind the attack, but since then, we have had more attacks with even higher severity.
To begin with, GNU Savannah, the FSF’s collaborative software development system, was hit by a massive botnet controlling about five million IPs starting in January. As of this writing, the attack is still ongoing, but the botnet’s current iteration is mitigated. The goal is likely to build an LLM training dataset. We do not know who or what is behind this.
Furthermore, gnu.org and ftp.gnu.org were targets in a new DDoS attack starting on May 27, 2025. Its goal seems to be to take the site down. It is currently mitigated. It has had several iterations, and each has caused some hours of downtime while we figured out how to defend ourselves against it. Here again, the goal was likely to take our sites down and we do not know who or what is behind this.
In addition, directory.fsf.org, the server behind the Free Software Directory, has been under attack since June 18. This likely is an LLM scraper designed to specifically target Media Wiki sites with a botnet. This attack is very active and now partially mitigated.
As we developed programs to identify IP addresses belonging to the botnet, they sometimes misidentified legitimate user’s IP addresses. We’ve removed them from the list of DDoS IP addresses and improved our defenses to be more precise. If you do not have access to gnu.org right now, please send us an email at sysadmin@fsf.org with your IP address and we will look into it. If you are having trouble with a VPN (virtual private network), try switching exit nodes and skip writing us – we know our attackers use VPNs, which leads us to block the ones they are using.
More recently, automated software build systems have become an issue for us. These usually go by the non-obvious term CI/CD, which stands for “continuous integration or continuous deployment.” They send automated requests to check for new code on Savannah in order to rebuild their software. They often send far more requests than is necessary, which looks and acts like a DDoS attack even though it is not intended to be. The CI/CD tooling does not typically have contact information labeling their traffic, so we do not have a way to contact them if there is a problem outside of banning their addresses or sending abuse reports if we can find a place to send them. We had to block some of these IP addresses, which often prompts them to search for better ways to accomplish the same goals.
On top of all of that, we have our run-of-the-mill standard crawlers, SEO (search engine optimization) crawlers, crawlers pretending to be normal users, crawlers pretending to be other crawlers, uptime systems, vulnerability scanners, carrier-grade network address translation, VPNs, and normal browsers hitting our sites. It is taxing for our sites and for our team of staff and volunteers, since we have to figure out a specific defense approach for each attack. Some of the abuse is not unique to us, and it seems that the health of the web has some serious problems right now.
When you visit a website, it might send your browser one or more JavaScript programs. These JavaScript programs are usually proprietary. We explain this more in “The JavaScript Trap.” If a website sends you a free JavaScript program, you can develop a modified version, share that with other people so they can benefit, and you can configure your browser to run your modified version instead of what the website sends. But some JavaScript programs are malware, which do things like spy on you, and the only modification any user would want is to stop it from ever running.
Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program’s calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom. With your support, we can continue to put these principles into practice.
Even though we are under active attack, gnu.org, ftp.gnu.org, and savannah.gnu.org are up with normal response times at the moment, and have been for the majority of this week, largely thanks to hard work from the Savannah hackers Bob, Corwin, and Luke who’ve helped us, your sysadmins. We’ve shielded these sites for almost a full year of intense attacks now, and we’ll keep on fighting these attacks for as long as they continue.
Thank you for being an associate member and for joining us in our crucial work to guard user freedom and defy Big Tech’s dystopia! Your support means so much to us. Can you convince a friend to become an FSF associate member as well? If you use your referral code (i.e. the member ID on your dashboard) we will send you a wooden GNU Head sticker in return. Alternatively, you can now sponsor a membership for someone who can’t afford to buy one themselves. With your support, we can defy the dystopia Big Tech is trying to bring on us.
Thank you for supporting the tech team!
Happy hacking,
Michael McMahon & Ian Kelling Your FSF Systems Administrators – Interested in helping us expand our reach?
Read our Privacy Policy: https://www.fsf.org/about/free-software-foundation-privacy-policy.
Sent from the Free Software Foundation,
31 Milk Street # 960789 Boston, Massachusetts 02196 United States
Markdown file for this page: https://wxie.codeberg.page/blog/Our-small-team-vs-millions-of-bots.md
Subscribe to RSS for this site
This HTML page was generated by the Untitled Static Site Generator.