Setuptools, a boring, ever-present Python package, necessary for billions of builds around the world, suddenly broke for millions of devs yesterday.

It's 9:10 AM, Silicon Valley time. andy-maier, posts on setuptools's GitHub Issues about how setuptools version 78.0.1 broke his pip install ansible-vault, given the following error:

setuptools.errors.InvalidConfigError: Invalid dash-separated key
'description-file' in 'metadata' (setup.cfg), please use the
underscore name 'description_file' instead.

At around the same time, I noticed my CI pipeline was failing on the poetry install step, with the same error - an error that locally didn't occur, so I was ready to admit the issue I was facing was going to be a long one to diagnose and fix. What could this error mean? Then, I myself came to setuptools's GitHub Issues, and stumbled upon andy-maier's thread. That's when I quickly realized:

  1. The error was simple to fix.
  2. I can't be the one fixing it (not the simple way, at least).
  3. The rest of my working day was ruined and would be spent looking into this.

So, this is how a large swath of Python developers came to realize just how fragile the Python build ecosystem, and, maybe, some package maintainers' ego, really can be.

Some background

In Python builds, pip pulls in the latest version of setuptools, and builds your package/project in an isolated build where the setuptools pip uses is not the same as the one you may have for your own environment. From the PEP 518 section in pip's documentation:

When making build requirements available, pip does so in an isolated environment. That is, pip does not install those requirements into the user’s site-packages, but rather installs them in a temporary directory which it adds to the user’s sys.path for the duration of the build. This ensures that build requirements are handled independently of the user’s runtime environment. For example, a project that needs a recent version of setuptools to build can still be installed, even if the user has an older version installed (and without silently replacing that version). In certain cases, projects (or redistributors) may have workflows that explicitly manage the build environment. For such workflows, build isolation can be problematic. If this is the case, pip provides a --no-build-isolation flag to disable build isolation. Users supplying this flag are responsible for ensuring the build environment is managed appropriately.

So, if hypothetically, the latest version of setuptools breaks your setup, there's really no way to way to circumvent this bad setuptools version unless you override the setuptools version to be different, and disable build isolation through the flag.

What happened

The project maintainers finally decided that enough was enough, setuptools would have to make a drastic change to address a very important issue, ending a 4 year wait with the release of version 78.x:

Setuptools no longer accepts options containing uppercase or dash characters in setup.cfg. Please ensure to write the options in setup.cfg using the lower_snake_case convention (e.g. Name => name, install-requires => install_requires). This is a follow-up on deprecations introduced in v54.1.0 (see #1608) and v54.1.1 (see #2592).

Yes, you read that right - they decided a hyphen was more important than other people's time and energy. This is, of course, demonstrably a bad design choice - the problem isn't even that nobody wants to update a setup.cfg file for no good reason, it's that most packages don't have a maintainer at all and don't get updated at all, let alone for such a non-value-add change: being picky about hyphens versus underscores.

Power-tripping, and falling

The people behind setuptools somehow agreed that, because this was 4 years in the making, it would be okay - surely everyone cares about lower_snake_case and making the code all pretty. It's one thing to care about formatting, which I do, another is to force others to care, in the shape of broken builds and killed time.

In fact, it was a pre-meditated murder, as a public thread, 2 weeks ago, clearly shows. One of the setuptools people says:

I'm inclined to say we should do it, even though it will cause some disruption. The only other options I can think of are:

  • Fail intermittently, so users can retry and succeed but also get the signal that something is broken and start complaining to the offending project.
  • During the deprecation warning, sleep for some time, giving users a chance to ponder their life choices and maybe investigate the cause, motivating complaints to the offending project.
  • Develop a more sophisticated signal from deprecations in build backends to the offending projects.
  • Roll back the deprecations and just accept the status quo.

The first three are really about the deprecation strategy of this project and not this specific deprecation. Given that to date, the deprecation process we have in place is the best process available, I'm inclined to just push forward with it, though I'm open to delay these depreciations if we think the churn or disruption is too great.

The first suggestion is downright macabre - software isn't just a toy you wield for ego and power, people expect things to work, you shouldn't randomly have some piece of code fail for no reason. Ignoring the non-sensical "sleep for some time" suggestion and dismissing the overly vague third, the obvious, glaring, stupifyingly simple fourth option is the best: don't deprecate what doesn't need to be deprecated. Sometimes, the best thing you can do is nothing.

To make matters even worse, they knew this would make a big fuss:

That said, I'm leaning toward option 1, get some signal out there to non-compliant projects, and then if the disruption is large, rollback and fallback to option 2 (now with the affected projects having visibility to the issue).

According to h0rv, a GitHub search for description-field yielded 12k repositories - roughly as many mentions to the word "yank" in related threads - in fact, the packages affected mentioned by users by name were too many, too many to tell.

The solution?

Yanking the forsaken setuptools version, of course! After hundreds of public comments, asking for a yank of the offending version, they finally did, and also released 78.0.2, reverting the preposterous change, roughly around 2 PM, 5 whole hours of setuptools being effectively down for a lot of devs. All they had to do was yank 78.x and think carefully.

The setuptools maintainers suggested various workarounds, each of which the community shot down, in favor of the obvious. All this... for a hyphen?

However, be warned, for the prophets foretell that the hyphen shall fall again!... per the snarky-toned 78.0.2 changelog:

Postponed removals of deprecated dash-separated and uppercase fields in setup.cfg. All packages with deprecated configurations are advised to move before 2026.

See you next year!