Understanding the Linux Backdoor: Implications for Open Source [When Penguins Cry]

so I don't want to scare you but did you

know that somebody checked a back door

into Linux that allows the attacker to

log into any Linux system I'll tell you

about that and I'll also tell you about

the time somebody tried to check a back

door into Ms Doss and did it from my

office that's a cool

[Music]

story hey I'm Dave welcome to my shop

I'm Dave plumber retired operating

systems engineer from Microsoft going

back to the MSDS and Windows 95 days and

today we're going to talk about the new

SSH vulnerability back door that's been

inserted into Linux what it means how it

works and what the implications are and

whether open source is any better off

than closed source and these types of

scenarios first off obviously I'm not a

Linux developer I mean I write code for

Linux but I've never contributed at

least not since about 1993 when I

checked in a bugfix to was it Minix I

think it was early Linux um or email the

I should say but suffice to say I'm not

a Linux contributor or developer so

while I may not be a Linux expert that's

probably good for you right now cuz I

have to explain things in terms that I

understand which means you will likely

understand them as well so before we

look at what happened first what are the

implications what does this back door do

somebody checked in some code and it

looks for a private key and it has a

public key built into SSH and what are

the implications well to put it simply

catastrophic had this not been detected

as early as it was it would have been

truly catastrophic for Linux because

let's say that this this is a nation

state what they've done is they've built

a backdoor private key into SSH that

anybody with the public key or actually

vice versa they put the public key in

the product and then they hold the

private key anybody who holds that

private key and only that person nobody

else can use this exploit I should be

clear about that only if you hold the

private key which the original person

who inserted the back door does then you

can connect to any Linux server with

this back door inserted in fact you

don't need a password you just need to

connect by SSH and the SSH server will

say hey you've got the right private key

in your metadata or whatever extra data

is sent along with login request it

matches that logs you right in basically

not only a completely open door it's

even better than that because it gives

you root access

I that side of it's a little more

technical but suffice to say the result

is that you get unfettered access to any

Linux system in the world that you want

that's running this exploit and you get

rud access so how' they do it well what

they did is they checked code into XZ

utils and when I'm saying they I'm

talking about one of the main developers

of the XZ utils the good news is this is

in 5.60 and 5.61 versions of the XD

utils and that's pretty much bleeding

edge it did make it into a few very

early drops of dros but not it's not

going to be like in 20.04 won't be

picked up immediately or anything into

there so if you're not running on the

bleeding edge long story short you're

probably

safe but surely you can't put a price on

your login safety I wouldn't have

thought so either but here we are so

they checked code into xzu tills that

long story short bakes the public key

into SSH and then accepts connections

from anybody with the private key but

wouldn't that be obvious it seems like

this should be obvious and easy to

detect just by reviewing the source code

change logs but in fact it's not because

they were very clever about how they did

it they didn't modify the source code at

all instead they injected encrypted and

compressed binary data that is decrypted

and then inserted into the build process

via the make file for the XZ tills if I

understand it correctly but this means

there's no changes to the code so

anybody reviewing the source code

changes doesn't see any change and who

reads make file changes I mean they're

pretty obtuse and opaque to begin with

and if you have any idea what a make

file is doing it's either a simple make

file or you're having a really good day

because a lot of make files are pretty

complex and hard to follow once you get

to a certain complexity of a project so

at this point we've got a back door in

the Linux source code and all we have to

do or all they have to do is wait for it

to be propagated through out all the

builds and dros and trickle down

everybody and if it doesn't get caught

before that boom hey by the way one

thing I've noticed lately on YouTube is

that people are like bang ending their

videos immediately and I guess this must

appease the algorithm in some way so I'm

going to do that today which means if

I'm going to tell you the stuff that I

tell you at the end of every episode I

have to tell you now like halfway

through so here it is I'm mostly in this

for the subs and likes so if you're

finding today's episode to be any

combination of informative or

entertaining please leave me one of each

before you go today and if you or

anybody you know is on the Spectrum then

please check out the free sample of my

book on Amazon where I tell you

everything I know about autism that I

wish I'd know in way back then so who

caught it

Microsoft well not really Microsoft guy

that works for Microsoft also happens to

do benchmarking and so on on the postgis

database and he was running his tests

and he noticed that the SSH logins were

taking a lot longer than they used to

and this happened all of a sudden so he

dug into it and he's the hero that found

the information and the back door and

published it and let us all know about

it so had it not been for that this

would have likely been uncaught now of

course this promps the bigger question

of which is more protective against this

type of back door an open source system

or a closed Source system and I'd say in

this case the source code actually isn't

all that relevant of course it's open

but so is the make file was anybody

really scrutinizing the make file and if

they had would they have caught it if

they looked at it carefully well nobody

even at Microsoft is probably hand

reviewing and scrutinizing the make

build code changes unless they cause a

problem if they slow things down they're

going to take a look at it but if you

make changes to the make file and the

binary pops out then everything is cool

it's probably not going to get that much

scrutiny unlike a change to the actual

source code that's going to go through

several layers of review and approval

and again I've been gone for Microsoft

20 years so I'm talking about how we did

it back 20 years ago which was probably

pretty sloppy and lazy and horrible

compared to what they're doing now but

here's the beauty of a regimental

commercial release with something like

Windows NT you don't just check in code

and then wait for it to show up in the

build it goes through a lot of processes

in between first there's a build lab

that picks up just your change build

your change independently builds a build

with your change kicks that out and it

goes through basic build and when all

the changes are collected for the day

that build will go through bvts or basic

verification tests and that's going to

check to make sure that all the normal

stuff works the smoke tests pass the

build boots does everything correctly

and that the performance is not

seriously degraded from previous builds

and that's how it would have been caught

early at Microsoft because the

significant impact to the login chain

would have caused the regression in when

log on if it were you know the

equivalent component in Windows and that

would have been detected immediately now

that's not to say that every such change

would be so it's not a golden guarantee

or anything but the fact that any change

to the product goes through both human

manual review of the textual changes to

the source code and then through a bunch

of verification tests and benchmarks and

several other verification tests that

make sure the functionality and the

performance hasn't regressed that

catches a lot of things it still won't

catch everything be clear on that so

there is no Panacea now the other thing

is when you're checking into code say

I'm checking into the heat manager well

the guy who write wrote The Heat manager

and the person who maintains the heat

manager is going to very carefully check

my changes when you get something like

exz utils I think there was only two

people working on it so did the other

person really vet these changes very

carefully or did lonus you know get out

of his computer and check every line I

doubt that so who is actually vetting

these changes as they go into the build

and the answer is I don't know because

I'm not involved in that process so I

can't really speak to what goes on

behind the scenes in Linux I can only

tell you what used to happen in Windows

how that you extrapolate that forward

and assume they're more careful now and

the protective aspects of that now the

difference is if it's in a part of Linux

that a lot of people care about a lot of

people are going to look at it but if

it's in a part of Linux that a lot of

people don't care about like xzu tills

then it's probably not going to get the

scrutiny that it deserves especially

when you see the implications of how

this change manifests so now a bit of my

own back door stories

when I was an intern in 1993 I was in MS

DOS working on MS DOS 6.2 where I worked

on Smart drive and highe and some of the

compression engine relocation disc copy

setup whole bunch of stuff and there was

a guy in my office that was also an

intern so we shared an office because we

were just interns and he decided that he

was working I believe on the copy

command and he took the opportunity to

check not a back door per se but a

special command line switch so instead

of you know how with x copy SLS we'll do

folder

well with copy he added it so that slart

which is like asky 254 I forget the

actual OEM code page at this moment but

whatever it was he used a special

graphic character and he made that a

command line switch and it would just

print I love sex over and over and over

and over and over and it would print

that out I didn't know anything about

this and then after I'm done my

internship I get a call from my manager

at the time who calls me at home and

says yeah so what did you know about

this and the answer was I knew nothing

and uh naturally they did not invite get

the guy back it was caught in the manual

source code Change review that I spoke

of earlier or however they caught it and

it did not look good for him I don't

know what the repercussions were I don't

know what they did I wasn't involved all

I know is that they took me at my word

that I wasn't involved and thankfully

for that because I want up working on

Microsoft for a long time and I don't

know if I would have hired me after that

little incident in my office did I

really not know about it I didn't so now

in all my time at Microsoft and in the

two decades since over this course of 30

years of Windows and Ms Doss to my

knowledge there has never been a back

door inserted into Windows now it's

possible that one has and it's just

never been found but I would assume that

by now there would be some exploit that

would take advantage of it is that

diligence process Superior coding or

pure luck well I'm going to guess a lot

of it's luck but a lot of it is process

doesn't mean the developers are any

smarter it does mean that people have

day jobs where they are assigned to look

at code changes in other people's

products whether they want to or not

then they are responsible for anything

that happens as a result of those

changes and that makes people pretty

honest and as for the people that aren't

honest there are processes and checks

and balances to make sure they don't do

a lot of harm thanks for joining me see

you