|
Xealous posted:I’ve got a few hundred stories like this keep em coming
|
# ? Aug 23, 2018 16:46 |
|
|
# ? Apr 26, 2024 04:15 |
|
infernal machines posted:why do the cars run a cluster of ubuntu vms? used to be centos 6 and Ruby on Rails. I haven’t worked there in 3 years, but last I heard it hadn’t changed much for s and x. model 3 uses newer tech, but still based out of a single Datacenter
|
# ? Aug 23, 2018 16:51 |
|
oh, woah, i read that post wrong at first. for some reason i read it as "almost caused the entire fleet" not "caused almost the entire fleet" holy poo poo lmbo that's even more amazingCat Face Joe posted:keep em coming dear god please yes
|
# ? Aug 23, 2018 16:58 |
|
also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar
|
# ? Aug 23, 2018 16:59 |
|
Farmer Crack-rear end posted:also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar appreciate the offer, I mostly browse and don’t post because I’m a boring computer toucher
|
# ? Aug 23, 2018 17:06 |
|
some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line
|
# ? Aug 23, 2018 17:10 |
|
Xealous posted:some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line incredible
|
# ? Aug 23, 2018 17:11 |
|
Xealous posted:some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line can't you flash the storage before its installed in a car?
|
# ? Aug 23, 2018 17:17 |
|
Xealous posted:some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line wh... why can't they pre-image the storage media?? e:f,b
|
# ? Aug 23, 2018 17:17 |
|
Xealous posted:some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line
|
# ? Aug 23, 2018 17:22 |
|
hobbesmaster posted:can't you flash the storage before its installed in a car? yes and no. the firmware update process in a car is complicated because you have a bunch of dumb components hanging off of CAN or LIN and they have to updated in very specific order and sometimes you have to retry 10s of times to get it to take. ( gently caress you Bosch). Tesla never bothered to flash those things ahead of time before assembly so that gets done the first time as it rolls down the line. the infotainment system and gateway arbitrate that stuff. typically any update that tuned voltages becomes a one way - no downgrade is possible without frying something
|
# ? Aug 23, 2018 17:23 |
|
hobbesmaster posted:can't you flash the storage before its installed in a car? courage
|
# ? Aug 23, 2018 17:24 |
|
Xealous posted:Tesla never bothered to flash those things ahead of time before assembly this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down
|
# ? Aug 23, 2018 17:28 |
|
hobbesmaster posted:this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl
|
# ? Aug 23, 2018 17:31 |
|
i get that they're a tech company, not a manufacturer, but who the gently caress designed that process and why at no point did anyone review, benchmark, then scrap it?
|
# ? Aug 23, 2018 17:34 |
|
like, for all the lols @ tesla, have they literally never heard of a process engineer?
|
# ? Aug 23, 2018 17:34 |
|
Xealous posted:they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl uh we literally do the same thing; well, yocto images and python tesla isn't the first to solder down SOMs running embedded linux and a bunch of MCUs hanging off an i2c/canbus/whatever line
|
# ? Aug 23, 2018 17:36 |
|
infernal machines posted:like, for all the lols @ tesla, have they literally never heard of a process engineer? like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something
|
# ? Aug 23, 2018 17:37 |
|
infernal machines posted:i get that they're a tech company, not a manufacturer, but who the gently caress designed that process and why at no point did anyone review, benchmark, then scrap it? they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken
|
# ? Aug 23, 2018 17:43 |
|
Endless Mike posted:they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken exactly this. we never really had time to address critical issues and were constantly short on staff because people were quitting or they just wouldn't give candidates competitive offers. this is why you hear about people burning out - they've managed to chase everyone away
|
# ? Aug 23, 2018 17:46 |
|
Xealous posted:like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something
|
# ? Aug 23, 2018 17:46 |
|
Xealous posted:like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something
|
# ? Aug 23, 2018 17:49 |
|
more fun facts: the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again. the logs themselves are written in a binary ring buffer format and when they come in they used to end up in a giant 700TB single mysql database after they were expanded. all of production after-sales service and engineering relies on that single log interpretation system which ran on centos 5 and python 2.4 until hbase/hadoop and friends were brought in. the supercharger system uses ssh dss keys to "vpn" back to the datacenter to a single server over 2G wireless with very limited resources. the connection is essentially simplex for various reasons so getting data to and from the supercharger is usually a 1KB/s operation unless that site has had connection aggregation done. at one point i looked at the system and to pull data out for analysis, somebody had written a bash script that was printf'ing in a for loop across ~5k devices. it would usually take about 3 days to do a successful firmware update on any single supercharger. we once patched openssl to ignore client cert expiry because somebody forgot to create a process to update keys in the field and all the customer cars started falling offline because their certs had expired. the quick and dirty was to just patch openssl quickly and make openvpn on the server side use that one while we created those processes for about 2 weeks.
|
# ? Aug 23, 2018 17:59 |
|
2G is dead so i wonder what happened to cars that used that
|
# ? Aug 23, 2018 18:01 |
|
Peeny Cheez posted:With "no fault of their own" being the usual "proposing a process that isn't fragile and prone to error but will take longer than X promised to Y who promised it to Z and hurting the feelings of V and W, who are responsible for the monstrosity in the first place and have the ear of management"? yep that was 90% of it. most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do poo poo like add farting unicorns
|
# ? Aug 23, 2018 18:03 |
|
infernal machines posted:i get that they're a tech company, not a manufacturer, lol
|
# ? Aug 23, 2018 18:05 |
|
hobbesmaster posted:uh we literally do the same thing; well, yocto images and python they aren't the first - for what we were doing at the time it made sense and helped us get the program off the ground quickly. lots of room for improvement and in 8 years, they should have done so. my issue was the fact that the systems doing the flashing were running the yocto images and perl and the guy writing the perl was also responsible for writing the thing that actually updates the car. that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot
|
# ? Aug 23, 2018 18:09 |
|
Xealous posted:that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot
|
# ? Aug 23, 2018 18:12 |
|
hobbesmaster posted:2G is dead so i wonder what happened to cars that used that not sure. anything using the "old gateway" would need to be replaced - i brought that up years ago. roadster, supercharger and rav4 all used that standard (though toyota pulled all the wireless connectivity from the board later on, wisely)
|
# ? Aug 23, 2018 18:13 |
|
i am SO GLAD your nda expired
|
# ? Aug 23, 2018 18:17 |
|
graph posted:i am SO GLAD your nda expired 99% of what i'm talking about is "public" anyway. tesla isn't encrypting their firmware and it's really easy to glean information from the vpn with a packet cap because nothing inside the vpn (was) encrypted. dumping tegra 3 model s and x is trivial and tesla's cars are nowhere near as secure as they'd have you believe. for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.
|
# ? Aug 23, 2018 18:25 |
|
Xealous posted:for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit. goldmine
|
# ? Aug 23, 2018 18:33 |
|
There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models. Does that sound plausible to you?
|
# ? Aug 23, 2018 18:39 |
|
while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root. more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows. autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.
|
# ? Aug 23, 2018 18:40 |
|
Sagebrush posted:There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models. yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around. fart-powered cars fucked around with this message at 18:47 on Aug 23, 2018 |
# ? Aug 23, 2018 18:44 |
|
Xealous posted:while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root. are these all done over cellular? the network itself provides a lot of security in that case at least. of course until someone gets your password to control center/command center
|
# ? Aug 23, 2018 18:49 |
|
Xealous posted:
wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz?
|
# ? Aug 23, 2018 18:49 |
|
Xealous posted:yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.
|
# ? Aug 23, 2018 18:50 |
|
Lutha Mahtin posted:wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz? pretty much. many quit in protest of the test, others didn't pass because they were harness people working in cad programs and not actually programmers. even HR, who at the time was run by the worst conspirator rear end in a top hat ( https://en.wikipedia.org/wiki/Arnnon_Geshuri ) told him to can it before they sank the program
|
# ? Aug 23, 2018 18:56 |
|
|
# ? Apr 26, 2024 04:15 |
|
Xealous posted:others didn't pass because they were harness people working in cad programs and not actually programmers. What in the literal gently caress?
|
# ? Aug 23, 2018 18:58 |