'Yesterday, packages.microsoft.com—the repository from which Microsoft
serves software installers for Linux distributions including CentOS,
Debian, Fedora, OpenSUSE, and more—went down hard, and it stayed down
for around 18 hours. The outage impacted users trying to install .NET
Core, Microsoft Teams, Microsoft SQL Server for Linux (yes, that's a
thing) and more—as well as Azure's own devops pipelines.
We first became aware of the problem Wednesday evening when we saw 404
errors in the output of apt update on an Ubuntu workstation with
Microsoft Teams installed. The outage is somewhat better-documented at
this .NET Core issue report on Github, with many users from all around
the world sharing their experiences and theories.
The short version is that the entire repository cluster that serves
all Linux packages for Microsoft was completely down—issuing a range
of HTTP 404 (content not found) and 500 (Internal Server Error)
messages for any URL—for roughly 18 hours. Microsoft engineer Rahul
Bhandari confirmed the outage roughly five hours after it was
initially reported, with a cryptic comment about the infrastructure
team "running into some space issues."
Eighteen hours after the issue was detailed, Bhandari said that the
mirrors were once again available—although with temporarily degraded
performance, likely due to cold caches. In this update, Bhandari said
that the original cause of the outage was "a regression in [apt
repositories] during some feature migration work that resulted in
those packages becoming unavailable on the mirrors."
We're still waiting for a comprehensive incident report, since
Bhandari's status updates provide clues but no real explanations. The
good news: We can confirm that packages.microsoft.com
is indeed up
once again, and it is serving packages as it should.'
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 577-5304