|
i used to work for tesla writing infotainment firmware and backend services - all of which runs in a single bottom tier Datacenter in a single location on the worst VMware deployment known to man. fun fact: a jenkins pipeline once caused almost the entire fleet to reboot loop for about an hour
|
#
¿
Aug 23, 2018 16:01
|
|
|
|
| # ¿ Apr 25, 2024 02:24 |
|
|
Farmer Crack-rear end posted:i model s and x use openvpn to talk to their backend. inside that backend there are metadata services that feed info to the system, one of those things being a ~20MB+ (generated by the worst erp system) json payload that describes supercharger poo poo for the map in the touchscreen. somebody was smart enough to do automated linting but forgot to validate against the custom parser the car runs which caused a segfault in the qt app that runs the ui, which in turn for a variety of reasons forces a reboot of that component. I think we clocked about 15 seconds before it read the file and faulted after boot. it was doing that for an hour before everyone panicked and got me and qa on the phone to fix it. i wrote a quick python/fabric script that ssh’d to as many cars as possible at a time to rm the file I’ve got a few hundred stories like this
|
|
#
¿
Aug 23, 2018 16:46
|
|
|
infernal machines posted:why do the cars run a cluster of ubuntu vms? used to be centos 6 and Ruby on Rails. I haven’t worked there in 3 years, but last I heard it hadn’t changed much for s and x. model 3 uses newer tech, but still based out of a single Datacenter
|
|
#
¿
Aug 23, 2018 16:51
|
|
|
Farmer Crack-rear end posted:also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar appreciate the offer, I mostly browse and don’t post because I’m a boring computer toucher
|
|
#
¿
Aug 23, 2018 17:06
|
|
|
some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the line
|
|
#
¿
Aug 23, 2018 17:10
|
|
|
hobbesmaster posted:can't you flash the storage before its installed in a car? yes and no. the firmware update process in a car is complicated because you have a bunch of dumb components hanging off of CAN or LIN and they have to updated in very specific order and sometimes you have to retry 10s of times to get it to take. ( gently caress you Bosch). Tesla never bothered to flash those things ahead of time before assembly so that gets done the first time as it rolls down the line. the infotainment system and gateway arbitrate that stuff. typically any update that tuned voltages becomes a one way - no downgrade is possible without frying something
|
|
#
¿
Aug 23, 2018 17:23
|
|
|
hobbesmaster posted:this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl
|
|
#
¿
Aug 23, 2018 17:31
|
|
|
infernal machines posted:like, for all the lols @ tesla, have they literally never heard of a process engineer? like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break something
|
|
#
¿
Aug 23, 2018 17:37
|
|
|
Endless Mike posted:they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken exactly this. we never really had time to address critical issues and were constantly short on staff because people were quitting or they just wouldn't give candidates competitive offers. this is why you hear about people burning out - they've managed to chase everyone away
|
|
#
¿
Aug 23, 2018 17:46
|
|
|
more fun facts: the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again. the logs themselves are written in a binary ring buffer format and when they come in they used to end up in a giant 700TB single mysql database after they were expanded. all of production after-sales service and engineering relies on that single log interpretation system which ran on centos 5 and python 2.4 until hbase/hadoop and friends were brought in. the supercharger system uses ssh dss keys to "vpn" back to the datacenter to a single server over 2G wireless with very limited resources. the connection is essentially simplex for various reasons so getting data to and from the supercharger is usually a 1KB/s operation unless that site has had connection aggregation done. at one point i looked at the system and to pull data out for analysis, somebody had written a bash script that was printf'ing in a for loop across ~5k devices. it would usually take about 3 days to do a successful firmware update on any single supercharger. we once patched openssl to ignore client cert expiry because somebody forgot to create a process to update keys in the field and all the customer cars started falling offline because their certs had expired. the quick and dirty was to just patch openssl quickly and make openvpn on the server side use that one while we created those processes for about 2 weeks.
|
|
#
¿
Aug 23, 2018 17:59
|
|
|
Peeny Cheez posted:With "no fault of their own" being the usual "proposing a process that isn't fragile and prone to error but will take longer than X promised to Y who promised it to Z and hurting the feelings of V and W, who are responsible for the monstrosity in the first place and have the ear of management"? yep that was 90% of it. most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do poo poo like add farting unicorns
|
|
#
¿
Aug 23, 2018 18:03
|
|
|
hobbesmaster posted:uh we literally do the same thing; well, yocto images and python they aren't the first - for what we were doing at the time it made sense and helped us get the program off the ground quickly. lots of room for improvement and in 8 years, they should have done so. my issue was the fact that the systems doing the flashing were running the yocto images and perl and the guy writing the perl was also responsible for writing the thing that actually updates the car. that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot
|
|
#
¿
Aug 23, 2018 18:09
|
|
|
hobbesmaster posted:2G is dead so i wonder what happened to cars that used that not sure. anything using the "old gateway" would need to be replaced - i brought that up years ago. roadster, supercharger and rav4 all used that standard (though toyota pulled all the wireless connectivity from the board later on, wisely)
|
|
#
¿
Aug 23, 2018 18:13
|
|
|
graph posted:i am SO GLAD your nda expired 99% of what i'm talking about is "public" anyway. tesla isn't encrypting their firmware and it's really easy to glean information from the vpn with a packet cap because nothing inside the vpn (was) encrypted. dumping tegra 3 model s and x is trivial and tesla's cars are nowhere near as secure as they'd have you believe. for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.
|
|
#
¿
Aug 23, 2018 18:25
|
|
|
while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root. more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows. autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.
|
|
#
¿
Aug 23, 2018 18:40
|
|
|
Sagebrush posted:There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models. yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around. fart-powered cars fucked around with this message at 18:47 on Aug 23, 2018 |
|
#
¿
Aug 23, 2018 18:44
|
|
|
Lutha Mahtin posted:wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz? pretty much. many quit in protest of the test, others didn't pass because they were harness people working in cad programs and not actually programmers. even HR, who at the time was run by the worst conspirator rear end in a top hat ( https://en.wikipedia.org/wiki/Arnnon_Geshuri ) told him to can it before they sank the program
|
|
#
¿
Aug 23, 2018 18:56
|
|
|
hobbesmaster posted:are these all done over cellular? the network itself provides a lot of security in that case at least. of course until someone gets your password to control center/command center they're done over cell and wifi, depending on how big or urgent the update is. the network does generally provide security in the transport sense - the backend systems are what worry me and tesla is a big enough target that a determined actor could gain control. i can only guess how the likes of NIO (a chinese e-car startup) runs their backend.
|
|
#
¿
Aug 23, 2018 19:00
|
|
|
CommieGIR posted:"We don't need QA/UAT, we just need to pound out a really insecure and buggy product! Jackpot!" we had firmware QA and they were great folks. for the lack of staff and timelines we had they were amazing people and the folks i worked directly with, save a few, were really talented. that said, it takes more than just firmware to QA something like a car that talks to a complex backend with continuous delivery into production
|
|
#
¿
Aug 23, 2018 19:04
|
|
|
the early days of tesla, post-roadster, early model s and the start of model x were good times - everyone was trying to prove the technology worked, we were innovating and making something that hadn't been done before. things really started to poo poo the bed around the time we pivoted from model 3 plans to shipping model x first. the falcon wing doors were _such a shitshow_. they ended up delaying the program almost a year, hence why model 3 basically skipped all the usual phases a car goes through for validation. i mean, come on - you have bumpers falling off in the rain, the interior is a disaster, there's no instrument cluster which takes your eyes off the road - this list just goes on.
|
|
#
¿
Aug 23, 2018 19:11
|
|
|
tesla basically runs their entire business like a just in time compiler only they don't treat warnings or errors as failures. most groups in the company don't cross-communicate so there's a lot of duplication of effort. i once got pulled into a meeting because a car burned down when it was attached to a supercharger and we didn't get a log out of the car. normally under some emergency circumstances the car will try to upload a log when it thinks poo poo has gone really badly, but in this particular case it was far enough away from a tower it had half 3G connection and had to upload a 30MB log via HTTPS POST. the car burned down before it even got to 10MB and the system was only designed for exponential backoff retries, not resumption of in-progress. elon was calm about it, but we had to justify why we never had time to address it - maybe it was because we were all busy making unsafe features work?
|
|
#
¿
Aug 23, 2018 22:48
|
|
|
also on the supercharger note - you can get blacklisted from using them if you charge on them all the time. that's because the supercharger bypasses the charging regulator boards and dumps directly into the pack at 300A/450v which creates a ton of wear on the battery. want to keep your range high? don't supercharge often.
|
|
#
¿
Aug 23, 2018 22:51
|
|
|
President Beep posted:do they define “too often”? algorithm-based now - the ai poo poo i was working on took into account a lot of factors to determine if you were abusing it before i left. the criteria takes into account the state of many components in the car, your driving patterns and other details. or it did anyway. not even sure that stuff is running still - they rotated projects in and out of existence pretty rapidly.
|
|
#
¿
Aug 23, 2018 22:59
|
|
|
GWBBQ posted:what is elon like when stuff goes wrong due to his idiotic micromanagement and big stupid ideas? he's never wrong. his "open door policy" was an invitation to catch you breaking rank.
|
|
#
¿
Aug 23, 2018 23:01
|
|
|
tesla was also in the news because they were doing cute poo poo like spinning up k8s clusters which had AWS IAM access to sensitive S3 buckets but wasn't ssl'd and the k8s mgmt api was available publicly. there were other teams running industrial control equipment with centos 7 an no hardening at all. there was one time where a canadian kid stole the domain and redirected emails and managed to take over slack and a bunch of other poo poo because the idiot IT team didn't hide the registrar information or use something like markmonitor. the car-side stuff at least did full mtls at the time so it was ok, but lol did that kid get a lot of info.
|
|
#
¿
Aug 23, 2018 23:13
|
|
|
C.H.O.M.E. posted:thats just what i want, the car manufacturer monitoring how i drive the car i own and deciding that features should be turned off after i have purchased it, that's a good feature. you have no idea. any connected car is ripe for data harvesting and you (the consumer) should expect it going forward. on that note, china has a law in place that mandates all electric cars send real time telemetry to their government servers - model s/x/3, NIO cars and any other electric car if they're driving already complies with that law to be road certified. don't be surprised if that becomes a mandate in other countries
|
|
#
¿
Aug 24, 2018 00:11
|
|
|
C.H.O.M.E. posted:thats ok my car is 15 years old and i will never buy a new one and the only thing i connect it to is my own butt i powered my old car with farts too
|
|
#
¿
Aug 24, 2018 00:23
|
|
|
for all the poo poo that went down at tesla, there were some positive aspects. everyone i worked with really cared about physical safety and we put a lot of effort into making sure the engineering was sound so nobody got hurt. if you subtract autopilot, and that's a big if, the car is generally well designed minus the fit and finish issues + interior, but i'd argue that's never been tesla's strong point anyway. the cars are fast, the 2013-2014 model s lines were really good, solid, basic cars. my last straw was the summon feature - i strongly believe a car you are not in, backing out on its own from a parking space with the current sensors is super dangerous. i was making jokes with the tesla expats when ol' musky launched his roadster into space that you could see the gaps in the fit and finish without a telescope
|
|
#
¿
Aug 24, 2018 01:48
|
|
|
infernal machines posted:what about this part: i always carry a window buster with me no matter the car i drive, but i see your point. i won't defend everything they do anyway - that seems legit to me. so minus 1
|
|
#
¿
Aug 24, 2018 02:08
|
|
|
redleader posted:btw thanks for posting these. they're amazing no problem - if you want to know anything specific (i won't violate any ndas, patents or internal arch) happy to oblige
|
|
#
¿
Aug 24, 2018 02:09
|
|
|
infernal machines posted:no worries. love your posts, i just thought defending tesla on the safety front was a bit strange i think i just got caught up in their collision safety. i dunno, most of us really tried our best to do things the safe way - i advocated for patching out shellshock and heartbleed across the company really hard but a lot of it fell on deaf ears for too long also advocated for switching out boards with something that had an hsm, doing hab stuff, encrypting the ssl keys the car has (they're on an unencrypted cf card, lol), mtls for everything on the backend, ipsec between components in the car, etc etc. i just gave up and quit
|
|
#
¿
Aug 24, 2018 02:17
|
|
|
Suspicious Dish posted:what size bed does grimes's ex-boyfriend sleep in on the factory floor i worked out of deer creek, never really saw him in the factory. rumor is he was stealing an office and sleeping on a couch *edit - i did see John McCain in deer creek once, that was a trip. showed him some of the stuff we were working on fart-powered cars fucked around with this message at 02:25 on Aug 24, 2018 |
|
#
¿
Aug 24, 2018 02:21
|
|
|
just remembered some bits of trivia * they took away our free snacks in deer creek and replaced them with lovely vendors * said vendors food poisoned people often enough osha or whatever the body is shut them down * people were so mad about the free cereal being gone they'd intra-office snail mail bowls of cereal from the factory and post pictures in slack * deer creek's parking got so bad (too many people, not enough space) they hired permanent valets * they were cited for the shitshow parking for fire safety violations (unconfirmed, but i believe it) * elon publicly being a shitbag to trans people * the first time we turned on real time telemetry for the dev fleet we caught somebody going 130mph over the san mateo bridge * it networking so bad the company had permanent 5~8% consistent packet loss between various places (like, next rack) * firmware git repo so large they had to mirror it (something like 2TB)
|
|
#
¿
Aug 24, 2018 02:32
|
|
|
Source4Leko posted:What would describe as being the most seriously hosed up thing you saw happen there? Define hosed up any way you want to. internal politics. the most toxic culture i have ever encountered, heard about or worked in i swear they selected for the shittiest people in silicon valley and made them managers
|
|
#
¿
Aug 24, 2018 02:34
|
|
|
Peeny Cheez posted:Can you confirm/deny no tailpipe but i wouldn't be surprised if he attempted the charging port
|
|
#
¿
Aug 24, 2018 02:41
|
|
|
Endless Mike posted:lmao $100k cars should not have issues with fit and finish that haven't been since the death of british leyland depending on when and what features you got (and if you got a marketing used car) they could go as low at $40k after incentives - but totally agree with you. fit/finish issues have been a thorn in their side forever the touchscreen is kind of a safety issue in that you have to look at it to touch it, stealing focus. tactile buttons for some functions would have been better
|
|
#
¿
Aug 24, 2018 02:47
|
|
|
Dumb Lowtax posted:Another safety issue at Tesla is that of being swatted by your CEO for pointing out that resin is being injected into the holes of unsafe lithium batteries and shipped into a bunch of customers' cars some friends of mine worked with the guy, the jury is still out on that one. ol' musky isn't totally paranoid - we did catch bad actors doing stuff and they were nailed to the wall. finding a real apt in your network can be some next level poo poo
|
|
#
¿
Aug 24, 2018 02:52
|
|
|
the firmware repo was that size if you take into account a huge company, many devices in the car at play and incremental updates to firmware across all those devices + branches for people to do work in. i contributed to that mess by policy, not by choice, but whatever. i'd imagine they'd be smart enough to move to something like git lfs so it isn't as much of a pain scale stuff: tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks i had to deal with marketing people sincerely asking me why we weren't going to run containers on the car in firmware. no, marketing, i don't care that the car would "update faster" or "features would release faster"  a web front-end (we'll say it's a cms that's php-based) that needed $500k in WAF bullshit just so we didn't get pwned every 5 minutes fragmented installs of splunk. i think i counted well over 20 installs for various departments before they finally hired a decent data scientist that cleaned it up so many random java, django, .net services from various places, more than i could count and i had to touch a _lot_ of them with firmware. ActiveRecord controlling way way _way_ too much. i consider this probably one of tesla's biggest scale problems - i don't think they actually know or can track exactly what they're running server side at all - so you end up with teams running vmware, nsx, k8s, openstack, hyper-v. a car that has a json parser implemented in bash 3 because <interpreted language> is dangerous in the car. there are some seriously magic shell scripts on that thing that probably 3 people in the company understand in full nodejs was a thing for a while but quickly broke down once we reached the 20k car mark - ended up replacing a bunch of that stuff with a Go variant
|
|
#
¿
Aug 24, 2018 05:41
|
|
|
GWBBQ posted:bets on whether the fire was due to incompetence, act of nature, or deliberately set? never attribute to malice what can more easily be explained by incompetence GWBBQ posted:not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on defective bottles and valves. do you think that attitude might have scared them into not saying anything? absolutely. taking advantage of the "open door policy" was the fastest way to lose your job at tesla and from what i'm told, spacex, being run by the same guy was no different. there is so much pressure to ship on time they push people to work 14 hour days, 7 days a week - i did that for a while before i just couldn't take it anymore and just accepted being marked down in employee review for being late
|
|
#
¿
Aug 24, 2018 05:57
|
|
|
|
| # ¿ Apr 25, 2024 02:24 |
|
|
infernal machines posted:i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end? imagine if all of your services were synchronous instead of async
|
|
#
¿
Aug 24, 2018 05:59
|
|