Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Cat Face Joe
Feb 20, 2005

goth vegan crossfit mom who vapes



Xealous posted:

I’ve got a few hundred stories like this

keep em coming

Adbot
ADBOT LOVES YOU

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

infernal machines posted:

why do the cars run a cluster of ubuntu vms?

used to be centos 6 and Ruby on Rails. I haven’t worked there in 3 years, but last I heard it hadn’t changed much for s and x. model 3 uses newer tech, but still based out of a single Datacenter

Farmer Crack-Ass
Jan 2, 2001

this is me posting irl
oh, woah, i read that post wrong at first. for some reason i read it as "almost caused the entire fleet" not "caused almost the entire fleet" holy poo poo lmbo that's even more amazing



Cat Face Joe posted:

keep em coming

dear god please yes

Farmer Crack-Ass
Jan 2, 2001

this is me posting irl
also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

Farmer Crack-rear end posted:

also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar

appreciate the offer, I mostly browse and don’t post because I’m a boring computer toucher

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR
some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line

Cat Face Joe
Feb 20, 2005

goth vegan crossfit mom who vapes



Xealous posted:

some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line

incredible

hobbesmaster
Jan 28, 2008

Xealous posted:

some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line

can't you flash the storage before its installed in a car?

Farmer Crack-Ass
Jan 2, 2001

this is me posting irl

Xealous posted:

some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line

wh... why can't they pre-image the storage media?? :psyduck:


e:f,b

Endless Mike
Aug 13, 2003



Xealous posted:

some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line

:stare:

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

hobbesmaster posted:

can't you flash the storage before its installed in a car?

yes and no. the firmware update process in a car is complicated because you have a bunch of dumb components hanging off of CAN or LIN and they have to updated in very specific order and sometimes you have to retry 10s of times to get it to take. ( gently caress you Bosch). Tesla never bothered to flash those things ahead of time before assembly so that gets done the first time as it rolls down the line. the infotainment system and gateway arbitrate that stuff. typically any update that tuned voltages becomes a one way - no downgrade is possible without frying something

OJ MIST 2 THE DICK
Sep 11, 2008

Anytime I need to see your face I just close my eyes
And I am taken to a place
Where your crystal minds and magenta feelings
Take up shelter in the base of my spine
Sweet like a chica cherry cola

-Cheap Trick

Nap Ghost

hobbesmaster posted:

can't you flash the storage before its installed in a car?

courage

hobbesmaster
Jan 28, 2008

Xealous posted:

Tesla never bothered to flash those things ahead of time before assembly

this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

hobbesmaster posted:

this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down

they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl

infernal machines
Oct 11, 2012

we have sealed ourselves away behind our money, growing inward, generating a seamless universe of self.
i get that they're a tech company, not a manufacturer, but who the gently caress designed that process and why at no point did anyone review, benchmark, then scrap it?

infernal machines
Oct 11, 2012

we have sealed ourselves away behind our money, growing inward, generating a seamless universe of self.
like, for all the lols @ tesla, have they literally never heard of a process engineer?

hobbesmaster
Jan 28, 2008

Xealous posted:

they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl

uh we literally do the same thing; well, yocto images and python

tesla isn't the first to solder down SOMs running embedded linux and a bunch of MCUs hanging off an i2c/canbus/whatever line

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

infernal machines posted:

like, for all the lols @ tesla, have they literally never heard of a process engineer?

like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something

Endless Mike
Aug 13, 2003



infernal machines posted:

i get that they're a tech company, not a manufacturer, but who the gently caress designed that process and why at no point did anyone review, benchmark, then scrap it?

they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

Endless Mike posted:

they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken

exactly this. we never really had time to address critical issues and were constantly short on staff because people were quitting or they just wouldn't give candidates competitive offers. this is why you hear about people burning out - they've managed to chase everyone away

Ellie Crabcakes
Feb 1, 2008

Stop emailing my boyfriend Gay Crungus

Xealous posted:

like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something
With "no fault of their own" being the usual "proposing a process that isn't fragile and prone to error but will take longer than X promised to Y who promised it to Z and hurting the feelings of V and W, who are responsible for the monstrosity in the first place and have the ear of management"?

prisoner of waffles
May 8, 2007

Ah! well a-day! what evil looks
Had I from old and young!
Instead of the cross, the fishmech
About my neck was hung.

Xealous posted:

like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR
more fun facts:

the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again. the logs themselves are written in a binary ring buffer format and when they come in they used to end up in a giant 700TB single mysql database after they were expanded. all of production after-sales service and engineering relies on that single log interpretation system which ran on centos 5 and python 2.4 until hbase/hadoop and friends were brought in.

the supercharger system uses ssh dss keys to "vpn" back to the datacenter to a single server over 2G wireless with very limited resources. the connection is essentially simplex for various reasons so getting data to and from the supercharger is usually a 1KB/s operation unless that site has had connection aggregation done. at one point i looked at the system and to pull data out for analysis, somebody had written a bash script that was printf'ing in a for loop across ~5k devices. it would usually take about 3 days to do a successful firmware update on any single supercharger.

we once patched openssl to ignore client cert expiry because somebody forgot to create a process to update keys in the field and all the customer cars started falling offline because their certs had expired. the quick and dirty was to just patch openssl quickly and make openvpn on the server side use that one while we created those processes for about 2 weeks.

hobbesmaster
Jan 28, 2008

2G is dead so i wonder what happened to cars that used that

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

Peeny Cheez posted:

With "no fault of their own" being the usual "proposing a process that isn't fragile and prone to error but will take longer than X promised to Y who promised it to Z and hurting the feelings of V and W, who are responsible for the monstrosity in the first place and have the ear of management"?

yep that was 90% of it. most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do poo poo like add farting unicorns

H.P. Hovercraft
Jan 12, 2004

one thing a computer can do that most humans can't is be sealed up in a cardboard box and sit in a warehouse
Slippery Tilde

infernal machines posted:

i get that they're a tech company, not a manufacturer,

lol

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

hobbesmaster posted:

uh we literally do the same thing; well, yocto images and python

tesla isn't the first to solder down SOMs running embedded linux and a bunch of MCUs hanging off an i2c/canbus/whatever line

they aren't the first - for what we were doing at the time it made sense and helped us get the program off the ground quickly. lots of room for improvement and in 8 years, they should have done so.

my issue was the fact that the systems doing the flashing were running the yocto images and perl and the guy writing the perl was also responsible for writing the thing that actually updates the car. that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot

hobbesmaster
Jan 28, 2008

Xealous posted:

that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot

:allbuttons:

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

hobbesmaster posted:

2G is dead so i wonder what happened to cars that used that

not sure. anything using the "old gateway" would need to be replaced - i brought that up years ago. roadster, supercharger and rav4 all used that standard (though toyota pulled all the wireless connectivity from the board later on, wisely)

graph
Nov 22, 2006

aaag peanuts
i am SO GLAD your nda expired

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

graph posted:

i am SO GLAD your nda expired

99% of what i'm talking about is "public" anyway. tesla isn't encrypting their firmware and it's really easy to glean information from the vpn with a packet cap because nothing inside the vpn (was) encrypted. dumping tegra 3 model s and x is trivial and tesla's cars are nowhere near as secure as they'd have you believe.

for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.

Roosevelt
Jul 18, 2009

I'm looking for the man who shot my paw.

Xealous posted:

for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.

goldmine

Sagebrush
Feb 26, 2012

There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models.

Does that sound plausible to you?

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR
while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root.

more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.

autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

Sagebrush posted:

There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models.

Does that sound plausible to you?

yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.

fart-powered cars fucked around with this message at 18:47 on Aug 23, 2018

hobbesmaster
Jan 28, 2008

Xealous posted:

while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root.

more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.

autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.

are these all done over cellular? the network itself provides a lot of security in that case at least. of course until someone gets your password to control center/command center

Lutha Mahtin
Oct 10, 2010

Your brokebrain sin is absolved...go and shitpost no more!

Xealous posted:


autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.

wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz?

ADINSX
Sep 9, 2003

Wanna run with my crew huh? Rule cyberspace and crunch numbers like I do?

Xealous posted:

yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.

:allears:

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR

Lutha Mahtin posted:

wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz?

pretty much. many quit in protest of the test, others didn't pass because they were harness people working in cad programs and not actually programmers. even HR, who at the time was run by the worst conspirator rear end in a top hat ( https://en.wikipedia.org/wiki/Arnnon_Geshuri ) told him to can it before they sank the program

Adbot
ADBOT LOVES YOU

Schadenboner
Aug 15, 2011

by Shine

Xealous posted:

others didn't pass because they were harness people working in cad programs and not actually programmers.

What in the literal gently caress?

:stare:

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply