Understanding the Linux Backdoor: Implications for Open Source [When Penguins Cry]
Summary
TLDRThe video discusses a recent SSH vulnerability back door found in Linux, inserted by a developer in XZ utils versions 5.60 and 5.61. This back door allows anyone with the private key to gain unfettered root access to any Linux system running the exploit. The video highlights the importance of security in open source and closed source systems, and shares a personal anecdote from the speaker's time at Microsoft, emphasizing the significance of thorough code review and build processes in preventing such security breaches.
Takeaways
- 🔑 A new SSH vulnerability back door has been discovered in Linux, which if exploited, allows unauthorized access to any Linux system without a password.
- 💡 The back door was inserted by checking in a code that searches for a private key, providing the attacker with root access if they possess the corresponding public key.
- 🛠️ The vulnerability was introduced in the XZ utils, specifically versions 5.60 and 5.61, which are considered bleeding edge and not widely adopted yet.
- 📈 A Microsoft employee who works on benchmarking discovered the SSH login delays and further investigation led to the detection of the back door.
- 🤔 The incident raises questions about the security of open-source versus closed-source systems and whether one is more protected against such vulnerabilities.
- 🔍 The back door was not easily detected because it was inserted as encrypted and compressed binary data, not as a change to the source code itself.
- 🚧 The complexity and obtuseness of make files made it difficult to spot the malicious changes, which were added via the build process.
- 🛑 The vulnerability would have been catastrophic if not detected early, as it could have been exploited by nation-states or other malicious actors.
- 📚 The speaker shares a personal anecdote from 1993 about an intern who inserted a prank command line switch into MS-DOS, highlighting the importance of code review.
- 🏛️ The speaker suggests that structured processes and checks and balances, as seen in commercial software development, can help catch and prevent such vulnerabilities from making it into production.
Q & A
What is the main topic of discussion in the video?
-The main topic of discussion is a new SSH vulnerability back door that was inserted into Linux, its implications, how it works, and a comparison between open source and closed source systems in terms of security.
What is the potential impact of the Linux SSH back door if it had not been detected early?
-If the Linux SSH back door had not been detected early, it would have been catastrophic for Linux systems as it would allow an attacker with the private key to gain unfettered root access to any Linux system running the exploit without needing a password.
How was the back door inserted into the Linux system?
-The back door was inserted by one of the main developers of XZ utils who checked in encrypted and compressed binary data that was decrypted and inserted into the build process via the make file, without modifying the source code directly.
Which versions of XZ utils were affected by the back door?
-The affected versions of XZ utils are 5.60 and 5.61.
How was the back door in Linux discovered?
-The back door was discovered by a Microsoft employee who was running benchmark tests on the PostGIS database and noticed that SSH logins were taking longer than usual. His investigation led to the discovery and publication of the back door.
What is the speaker's opinion on whether open source or closed source systems are better protected against back doors?
-The speaker believes that the source code itself is not as relevant as the processes and checks in place. While open source code is accessible for review, the actual protection against back doors depends on the scrutiny and verification processes, which may be more rigorous in closed source systems like Windows.
What was the speaker's experience with a back door in Microsoft DOS 6.2?
-During an internship at Microsoft, the speaker was unaware of a fellow intern who inserted a special command line switch in the copy command to print 'I love sex' repeatedly. This was caught in a manual source code review and the intern was not invited back.
How does the speaker suggest that such back doors can be caught in closed source systems?
-The speaker suggests that in closed source systems like Windows, there are multiple layers of review and verification tests that any change to the product must go through. This includes build lab testing, basic verification tests, and performance benchmarks, which can catch significant changes that could indicate the presence of a back door.
What is the speaker's conclusion on the effectiveness of processes in preventing back doors?
-The speaker concludes that while processes and checks and balances cannot guarantee catching every back door, they significantly increase the chances of detection due to mandatory reviews and tests, making people more honest and reducing the potential for harm.
What is the speaker's personal experience with code reviews in the past?
-The speaker shares that during his time at Microsoft, he was involved in code reviews where changes were carefully scrutinized, especially for critical components. He suggests that the level of scrutiny might vary depending on the popularity and关注度 of the part of the system being modified.
What is the speaker's final message to the viewers?
-The speaker thanks the viewers for joining him and encourages them to leave a subscription and a like if they found the episode informative or entertaining. He also promotes his book on autism for those who are on the spectrum or know someone who is.
Outlines
🔒 The Hidden Dangers of Backdoors in Linux and MSDOS
Dave Plummer, a retired operating systems engineer from Microsoft, shares alarming incidents of backdoors inserted into Linux and MSDOS. He details a recent discovery of a backdoor in Linux's SSH, allowing attackers unfettered root access to systems without needing a password. This vulnerability stems from malicious code inserted into XZ utils, which, if undetected, could have had catastrophic implications for Linux users worldwide. Dave contrasts this with his early days at Microsoft, recounting a colleague's attempt to insert a non-malicious 'easter egg' into MSDOS. Both stories underline the perpetual risk and vigilance required to safeguard operating systems from internal and external threats.
🔎 Open vs. Closed Source Security: A Comparative Analysis
Exploring the implications of the Linux SSH backdoor further, Dave Plummer debates the effectiveness of open versus closed source models in preventing such vulnerabilities. He argues that the visibility of source code, as in open source projects, does not necessarily guarantee security, citing the complexity of reviewing changes in make files and the overall scrutiny process. Drawing on his Microsoft experience, he outlines the rigorous review and testing processes in closed source development that might better catch such vulnerabilities. Dave shares a personal anecdote from his internship at Microsoft, illustrating the potential for mischief within development teams and highlighting the importance of oversight and review processes in software development.
🛡️ Lessons Learned and Closing Thoughts
In the final segment, Dave Plummer reflects on the broader implications of software vulnerabilities and the importance of vigilance in development processes, regardless of the open or closed source model. He emphasizes that while processes and checks are crucial for preventing unauthorized changes, a mix of luck and diligent oversight has historically kept major backdoors out of systems like Windows. Dave's insights offer valuable lessons on the complexities of software security and the ongoing need for comprehensive review mechanisms to protect against vulnerabilities.
Mindmap
Keywords
💡SSH vulnerability
💡Back door
💡Private key
💡Public key
💡XZ utils
💡Make file
💡Open source
💡Closed source
💡Benchmarking
💡Regression testing
💡Code review
Highlights
A new SSH vulnerability back door has been discovered in Linux systems.
The back door allows an attacker with the private key to log into any affected Linux system without a password.
The vulnerability provides root access, giving the attacker unfettered access to the system.
The back door was inserted by checking code into XZ utils, which is used in the build process of Linux.
The malicious code was inserted in versions 5.60 and 5.61 of XZ utils, affecting bleeding-edge builds.
The back door was not immediately obvious due to its clever insertion via encrypted and compressed binary data.
The vulnerability was detected by a Microsoft employee running benchmarks on the PostGIS database.
Open source versus closed source systems were discussed in terms of their susceptibility to such back doors.
The importance of rigorous code review and build processes in commercial releases was highlighted.
The potential protective nature of having multiple layers of review and testing in closed source systems was mentioned.
The speaker shares a personal anecdote from 1993 about an intern who inserted a prank command into MS-DOS.
The prank could have been caught by thorough manual source code review, which is more common in commercial software development.
The speaker asserts that throughout their career at Microsoft, no back doors were inserted into Windows.
The importance of processes, checks, and balances in software development to prevent malicious code insertion was emphasized.
The speaker concludes by appreciating the audience for their attention and participation.
Transcripts
so I don't want to scare you but did you
know that somebody checked a back door
into Linux that allows the attacker to
log into any Linux system I'll tell you
about that and I'll also tell you about
the time somebody tried to check a back
door into Ms Doss and did it from my
office that's a cool
[Music]
story hey I'm Dave welcome to my shop
I'm Dave plumber retired operating
systems engineer from Microsoft going
back to the MSDS and Windows 95 days and
today we're going to talk about the new
SSH vulnerability back door that's been
inserted into Linux what it means how it
works and what the implications are and
whether open source is any better off
than closed source and these types of
scenarios first off obviously I'm not a
Linux developer I mean I write code for
Linux but I've never contributed at
least not since about 1993 when I
checked in a bugfix to was it Minix I
think it was early Linux um or email the
I should say but suffice to say I'm not
a Linux contributor or developer so
while I may not be a Linux expert that's
probably good for you right now cuz I
have to explain things in terms that I
understand which means you will likely
understand them as well so before we
look at what happened first what are the
implications what does this back door do
somebody checked in some code and it
looks for a private key and it has a
public key built into SSH and what are
the implications well to put it simply
catastrophic had this not been detected
as early as it was it would have been
truly catastrophic for Linux because
let's say that this this is a nation
state what they've done is they've built
a backdoor private key into SSH that
anybody with the public key or actually
vice versa they put the public key in
the product and then they hold the
private key anybody who holds that
private key and only that person nobody
else can use this exploit I should be
clear about that only if you hold the
private key which the original person
who inserted the back door does then you
can connect to any Linux server with
this back door inserted in fact you
don't need a password you just need to
connect by SSH and the SSH server will
say hey you've got the right private key
in your metadata or whatever extra data
is sent along with login request it
matches that logs you right in basically
not only a completely open door it's
even better than that because it gives
you root access
I that side of it's a little more
technical but suffice to say the result
is that you get unfettered access to any
Linux system in the world that you want
that's running this exploit and you get
rud access so how' they do it well what
they did is they checked code into XZ
utils and when I'm saying they I'm
talking about one of the main developers
of the XZ utils the good news is this is
in 5.60 and 5.61 versions of the XD
utils and that's pretty much bleeding
edge it did make it into a few very
early drops of dros but not it's not
going to be like in 20.04 won't be
picked up immediately or anything into
there so if you're not running on the
bleeding edge long story short you're
probably
safe but surely you can't put a price on
your login safety I wouldn't have
thought so either but here we are so
they checked code into xzu tills that
long story short bakes the public key
into SSH and then accepts connections
from anybody with the private key but
wouldn't that be obvious it seems like
this should be obvious and easy to
detect just by reviewing the source code
change logs but in fact it's not because
they were very clever about how they did
it they didn't modify the source code at
all instead they injected encrypted and
compressed binary data that is decrypted
and then inserted into the build process
via the make file for the XZ tills if I
understand it correctly but this means
there's no changes to the code so
anybody reviewing the source code
changes doesn't see any change and who
reads make file changes I mean they're
pretty obtuse and opaque to begin with
and if you have any idea what a make
file is doing it's either a simple make
file or you're having a really good day
because a lot of make files are pretty
complex and hard to follow once you get
to a certain complexity of a project so
at this point we've got a back door in
the Linux source code and all we have to
do or all they have to do is wait for it
to be propagated through out all the
builds and dros and trickle down
everybody and if it doesn't get caught
before that boom hey by the way one
thing I've noticed lately on YouTube is
that people are like bang ending their
videos immediately and I guess this must
appease the algorithm in some way so I'm
going to do that today which means if
I'm going to tell you the stuff that I
tell you at the end of every episode I
have to tell you now like halfway
through so here it is I'm mostly in this
for the subs and likes so if you're
finding today's episode to be any
combination of informative or
entertaining please leave me one of each
before you go today and if you or
anybody you know is on the Spectrum then
please check out the free sample of my
book on Amazon where I tell you
everything I know about autism that I
wish I'd know in way back then so who
caught it
Microsoft well not really Microsoft guy
that works for Microsoft also happens to
do benchmarking and so on on the postgis
database and he was running his tests
and he noticed that the SSH logins were
taking a lot longer than they used to
and this happened all of a sudden so he
dug into it and he's the hero that found
the information and the back door and
published it and let us all know about
it so had it not been for that this
would have likely been uncaught now of
course this promps the bigger question
of which is more protective against this
type of back door an open source system
or a closed Source system and I'd say in
this case the source code actually isn't
all that relevant of course it's open
but so is the make file was anybody
really scrutinizing the make file and if
they had would they have caught it if
they looked at it carefully well nobody
even at Microsoft is probably hand
reviewing and scrutinizing the make
build code changes unless they cause a
problem if they slow things down they're
going to take a look at it but if you
make changes to the make file and the
binary pops out then everything is cool
it's probably not going to get that much
scrutiny unlike a change to the actual
source code that's going to go through
several layers of review and approval
and again I've been gone for Microsoft
20 years so I'm talking about how we did
it back 20 years ago which was probably
pretty sloppy and lazy and horrible
compared to what they're doing now but
here's the beauty of a regimental
commercial release with something like
Windows NT you don't just check in code
and then wait for it to show up in the
build it goes through a lot of processes
in between first there's a build lab
that picks up just your change build
your change independently builds a build
with your change kicks that out and it
goes through basic build and when all
the changes are collected for the day
that build will go through bvts or basic
verification tests and that's going to
check to make sure that all the normal
stuff works the smoke tests pass the
build boots does everything correctly
and that the performance is not
seriously degraded from previous builds
and that's how it would have been caught
early at Microsoft because the
significant impact to the login chain
would have caused the regression in when
log on if it were you know the
equivalent component in Windows and that
would have been detected immediately now
that's not to say that every such change
would be so it's not a golden guarantee
or anything but the fact that any change
to the product goes through both human
manual review of the textual changes to
the source code and then through a bunch
of verification tests and benchmarks and
several other verification tests that
make sure the functionality and the
performance hasn't regressed that
catches a lot of things it still won't
catch everything be clear on that so
there is no Panacea now the other thing
is when you're checking into code say
I'm checking into the heat manager well
the guy who write wrote The Heat manager
and the person who maintains the heat
manager is going to very carefully check
my changes when you get something like
exz utils I think there was only two
people working on it so did the other
person really vet these changes very
carefully or did lonus you know get out
of his computer and check every line I
doubt that so who is actually vetting
these changes as they go into the build
and the answer is I don't know because
I'm not involved in that process so I
can't really speak to what goes on
behind the scenes in Linux I can only
tell you what used to happen in Windows
how that you extrapolate that forward
and assume they're more careful now and
the protective aspects of that now the
difference is if it's in a part of Linux
that a lot of people care about a lot of
people are going to look at it but if
it's in a part of Linux that a lot of
people don't care about like xzu tills
then it's probably not going to get the
scrutiny that it deserves especially
when you see the implications of how
this change manifests so now a bit of my
own back door stories
when I was an intern in 1993 I was in MS
DOS working on MS DOS 6.2 where I worked
on Smart drive and highe and some of the
compression engine relocation disc copy
setup whole bunch of stuff and there was
a guy in my office that was also an
intern so we shared an office because we
were just interns and he decided that he
was working I believe on the copy
command and he took the opportunity to
check not a back door per se but a
special command line switch so instead
of you know how with x copy SLS we'll do
folder
well with copy he added it so that slart
which is like asky 254 I forget the
actual OEM code page at this moment but
whatever it was he used a special
graphic character and he made that a
command line switch and it would just
print I love sex over and over and over
and over and over and it would print
that out I didn't know anything about
this and then after I'm done my
internship I get a call from my manager
at the time who calls me at home and
says yeah so what did you know about
this and the answer was I knew nothing
and uh naturally they did not invite get
the guy back it was caught in the manual
source code Change review that I spoke
of earlier or however they caught it and
it did not look good for him I don't
know what the repercussions were I don't
know what they did I wasn't involved all
I know is that they took me at my word
that I wasn't involved and thankfully
for that because I want up working on
Microsoft for a long time and I don't
know if I would have hired me after that
little incident in my office did I
really not know about it I didn't so now
in all my time at Microsoft and in the
two decades since over this course of 30
years of Windows and Ms Doss to my
knowledge there has never been a back
door inserted into Windows now it's
possible that one has and it's just
never been found but I would assume that
by now there would be some exploit that
would take advantage of it is that
diligence process Superior coding or
pure luck well I'm going to guess a lot
of it's luck but a lot of it is process
doesn't mean the developers are any
smarter it does mean that people have
day jobs where they are assigned to look
at code changes in other people's
products whether they want to or not
then they are responsible for anything
that happens as a result of those
changes and that makes people pretty
honest and as for the people that aren't
honest there are processes and checks
and balances to make sure they don't do
a lot of harm thanks for joining me see
you
5.0 / 5 (0 votes)