Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
«2 »
  • Post
  • Reply
fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


i used to work for tesla writing infotainment firmware and backend services - all of which runs in a single bottom tier Datacenter in a single location on the worst VMware deployment known to man.

fun fact: a jenkins pipeline once caused almost the entire fleet to reboot loop for about an hour

Adbot
ADBOT LOVES YOU

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Farmer Crack-rear end posted:

i want need to know more

model s and x use openvpn to talk to their backend. inside that backend there are metadata services that feed info to the system, one of those things being a ~20MB+ (generated by the worst erp system) json payload that describes supercharger poo poo for the map in the touchscreen. somebody was smart enough to do automated linting but forgot to validate against the custom parser the car runs which caused a segfault in the qt app that runs the ui, which in turn for a variety of reasons forces a reboot of that component. I think we clocked about 15 seconds before it read the file and faulted after boot. it was doing that for an hour before everyone panicked and got me and qa on the phone to fix it. i wrote a quick python/fabric script that sshíd to as many cars as possible at a time to rm the file

Iíve got a few hundred stories like this

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

why do the cars run a cluster of ubuntu vms?

used to be centos 6 and Ruby on Rails. I havenít worked there in 3 years, but last I heard it hadnít changed much for s and x. model 3 uses newer tech, but still based out of a single Datacenter

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Farmer Crack-rear end posted:

also drat dude you want plat or a new av or anything, it pains me to see an '01 slummin' a trumptar

appreciate the offer, I mostly browse and donít post because Iím a boring computer toucher

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason canít talk to the PKI system, erp, or a ruby webapp it halts the line

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

can't you flash the storage before its installed in a car?

yes and no. the firmware update process in a car is complicated because you have a bunch of dumb components hanging off of CAN or LIN and they have to updated in very specific order and sometimes you have to retry 10s of times to get it to take. ( gently caress you Bosch). Tesla never bothered to flash those things ahead of time before assembly so that gets done the first time as it rolls down the line. the infotainment system and gateway arbitrate that stuff. typically any update that tuned voltages becomes a one way - no downgrade is possible without frying something

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the board is installed in anything if not before even being soldered down

they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perl

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

like, for all the lols @ tesla, have they literally never heard of a process engineer?

like everyone else who was smart they either quit or were fired through no fault of their own so what youíre left with are people fearing for their job who desperately donít want to change status quo for fear it will break something

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Endless Mike posted:

they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken

exactly this. we never really had time to address critical issues and were constantly short on staff because people were quitting or they just wouldn't give candidates competitive offers. this is why you hear about people burning out - they've managed to chase everyone away

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


more fun facts:

the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again. the logs themselves are written in a binary ring buffer format and when they come in they used to end up in a giant 700TB single mysql database after they were expanded. all of production after-sales service and engineering relies on that single log interpretation system which ran on centos 5 and python 2.4 until hbase/hadoop and friends were brought in.

the supercharger system uses ssh dss keys to "vpn" back to the datacenter to a single server over 2G wireless with very limited resources. the connection is essentially simplex for various reasons so getting data to and from the supercharger is usually a 1KB/s operation unless that site has had connection aggregation done. at one point i looked at the system and to pull data out for analysis, somebody had written a bash script that was printf'ing in a for loop across ~5k devices. it would usually take about 3 days to do a successful firmware update on any single supercharger.

we once patched openssl to ignore client cert expiry because somebody forgot to create a process to update keys in the field and all the customer cars started falling offline because their certs had expired. the quick and dirty was to just patch openssl quickly and make openvpn on the server side use that one while we created those processes for about 2 weeks.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Peeny Cheez posted:

With "no fault of their own" being the usual "proposing a process that isn't fragile and prone to error but will take longer than X promised to Y who promised it to Z and hurting the feelings of V and W, who are responsible for the monstrosity in the first place and have the ear of management"?

yep that was 90% of it. most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do poo poo like add farting unicorns

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

uh we literally do the same thing; well, yocto images and python

tesla isn't the first to solder down SOMs running embedded linux and a bunch of MCUs hanging off an i2c/canbus/whatever line

they aren't the first - for what we were doing at the time it made sense and helped us get the program off the ground quickly. lots of room for improvement and in 8 years, they should have done so.

my issue was the fact that the systems doing the flashing were running the yocto images and perl and the guy writing the perl was also responsible for writing the thing that actually updates the car. that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riot

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

2G is dead so i wonder what happened to cars that used that

not sure. anything using the "old gateway" would need to be replaced - i brought that up years ago. roadster, supercharger and rav4 all used that standard (though toyota pulled all the wireless connectivity from the board later on, wisely)

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


graph posted:

i am SO GLAD your nda expired

99% of what i'm talking about is "public" anyway. tesla isn't encrypting their firmware and it's really easy to glean information from the vpn with a packet cap because nothing inside the vpn (was) encrypted. dumping tegra 3 model s and x is trivial and tesla's cars are nowhere near as secure as they'd have you believe.

for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root.

more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.

autopilot had _really_ high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Sagebrush posted:

There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models.

Does that sound plausible to you?

yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.

fart-powered cars fucked around with this message at Aug 23, 2018 around 17:47

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Lutha Mahtin posted:

wait, do you mean the guy came in and cleaned out the department of people who couldn't pass his personal fizzbuzz pointer quiz?

pretty much. many quit in protest of the test, others didn't pass because they were harness people working in cad programs and not actually programmers. even HR, who at the time was run by the worst conspirator rear end in a top hat ( https://en.wikipedia.org/wiki/Arnnon_Geshuri ) told him to can it before they sank the program

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

are these all done over cellular? the network itself provides a lot of security in that case at least. of course until someone gets your password to control center/command center

they're done over cell and wifi, depending on how big or urgent the update is. the network does generally provide security in the transport sense - the backend systems are what worry me and tesla is a big enough target that a determined actor could gain control. i can only guess how the likes of NIO (a chinese e-car startup) runs their backend.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


CommieGIR posted:

"We don't need QA/UAT, we just need to pound out a really insecure and buggy product! Jackpot!"

we had firmware QA and they were great folks. for the lack of staff and timelines we had they were amazing people and the folks i worked directly with, save a few, were really talented. that said, it takes more than just firmware to QA something like a car that talks to a complex backend with continuous delivery into production

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


the early days of tesla, post-roadster, early model s and the start of model x were good times - everyone was trying to prove the technology worked, we were innovating and making something that hadn't been done before. things really started to poo poo the bed around the time we pivoted from model 3 plans to shipping model x first. the falcon wing doors were _such a shitshow_. they ended up delaying the program almost a year, hence why model 3 basically skipped all the usual phases a car goes through for validation. i mean, come on - you have bumpers falling off in the rain, the interior is a disaster, there's no instrument cluster which takes your eyes off the road - this list just goes on.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


tesla basically runs their entire business like a just in time compiler only they don't treat warnings or errors as failures. most groups in the company don't cross-communicate so there's a lot of duplication of effort.

i once got pulled into a meeting because a car burned down when it was attached to a supercharger and we didn't get a log out of the car. normally under some emergency circumstances the car will try to upload a log when it thinks poo poo has gone really badly, but in this particular case it was far enough away from a tower it had half 3G connection and had to upload a 30MB log via HTTPS POST. the car burned down before it even got to 10MB and the system was only designed for exponential backoff retries, not resumption of in-progress. elon was calm about it, but we had to justify why we never had time to address it - maybe it was because we were all busy making unsafe features work?

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


also on the supercharger note - you can get blacklisted from using them if you charge on them all the time. that's because the supercharger bypasses the charging regulator boards and dumps directly into the pack at 300A/450v which creates a ton of wear on the battery. want to keep your range high? don't supercharge often.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


President Beep posted:

do they define ďtoo oftenĒ?

algorithm-based now - the ai poo poo i was working on took into account a lot of factors to determine if you were abusing it before i left. the criteria takes into account the state of many components in the car, your driving patterns and other details. or it did anyway. not even sure that stuff is running still - they rotated projects in and out of existence pretty rapidly.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


GWBBQ posted:

what is elon like when stuff goes wrong due to his idiotic micromanagement and big stupid ideas?

he's never wrong. his "open door policy" was an invitation to catch you breaking rank.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


tesla was also in the news because they were doing cute poo poo like spinning up k8s clusters which had AWS IAM access to sensitive S3 buckets but wasn't ssl'd and the k8s mgmt api was available publicly. there were other teams running industrial control equipment with centos 7 an no hardening at all.

there was one time where a canadian kid stole the domain and redirected emails and managed to take over slack and a bunch of other poo poo because the idiot IT team didn't hide the registrar information or use something like markmonitor. the car-side stuff at least did full mtls at the time so it was ok, but lol did that kid get a lot of info.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


C.H.O.M.E. posted:

thats just what i want, the car manufacturer monitoring how i drive the car i own and deciding that features should be turned off after i have purchased it, that's a good feature.

you have no idea. any connected car is ripe for data harvesting and you (the consumer) should expect it going forward. on that note, china has a law in place that mandates all electric cars send real time telemetry to their government servers - model s/x/3, NIO cars and any other electric car if they're driving already complies with that law to be road certified. don't be surprised if that becomes a mandate in other countries

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


C.H.O.M.E. posted:

thats ok my car is 15 years old and i will never buy a new one and the only thing i connect it to is my own butt

i powered my old car with farts too

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


for all the poo poo that went down at tesla, there were some positive aspects. everyone i worked with really cared about physical safety and we put a lot of effort into making sure the engineering was sound so nobody got hurt. if you subtract autopilot, and that's a big if, the car is generally well designed minus the fit and finish issues + interior, but i'd argue that's never been tesla's strong point anyway. the cars are fast, the 2013-2014 model s lines were really good, solid, basic cars. my last straw was the summon feature - i strongly believe a car you are not in, backing out on its own from a parking space with the current sensors is super dangerous.

i was making jokes with the tesla expats when ol' musky launched his roadster into space that you could see the gaps in the fit and finish without a telescope

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

what about this part:



i always carry a window buster with me no matter the car i drive, but i see your point. i won't defend everything they do anyway - that seems legit to me. so minus 1

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


redleader posted:

btw thanks for posting these. they're amazing

no problem - if you want to know anything specific (i won't violate any ndas, patents or internal arch) happy to oblige

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

no worries. love your posts, i just thought defending tesla on the safety front was a bit strange

i think i just got caught up in their collision safety. i dunno, most of us really tried our best to do things the safe way - i advocated for patching out shellshock and heartbleed across the company really hard but a lot of it fell on deaf ears for too long

also advocated for switching out boards with something that had an hsm, doing hab stuff, encrypting the ssl keys the car has (they're on an unencrypted cf card, lol), mtls for everything on the backend, ipsec between components in the car, etc etc. i just gave up and quit

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Suspicious Dish posted:

what size bed does grimes's ex-boyfriend sleep in on the factory floor

i worked out of deer creek, never really saw him in the factory. rumor is he was stealing an office and sleeping on a couch

*edit - i did see John McCain in deer creek once, that was a trip. showed him some of the stuff we were working on

fart-powered cars fucked around with this message at Aug 24, 2018 around 01:25

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


just remembered some bits of trivia

* they took away our free snacks in deer creek and replaced them with lovely vendors
* said vendors food poisoned people often enough osha or whatever the body is shut them down
* people were so mad about the free cereal being gone they'd intra-office snail mail bowls of cereal from the factory and post pictures in slack
* deer creek's parking got so bad (too many people, not enough space) they hired permanent valets
* they were cited for the shitshow parking for fire safety violations (unconfirmed, but i believe it)
* elon publicly being a shitbag to trans people
* the first time we turned on real time telemetry for the dev fleet we caught somebody going 130mph over the san mateo bridge
* it networking so bad the company had permanent 5~8% consistent packet loss between various places (like, next rack)
* firmware git repo so large they had to mirror it (something like 2TB)

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Source4Leko posted:

What would describe as being the most seriously hosed up thing you saw happen there? Define hosed up any way you want to.

internal politics. the most toxic culture i have ever encountered, heard about or worked in

i swear they selected for the shittiest people in silicon valley and made them managers

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Peeny Cheez posted:

Can you confirm/deny



no tailpipe but i wouldn't be surprised if he attempted the charging port

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Endless Mike posted:

lmao $100k cars should not have issues with fit and finish that haven't been since the death of british leyland

depending on when and what features you got (and if you got a marketing used car) they could go as low at $40k after incentives - but totally agree with you. fit/finish issues have been a thorn in their side forever

the touchscreen is kind of a safety issue in that you have to look at it to touch it, stealing focus. tactile buttons for some functions would have been better

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


Dumb Lowtax posted:

Another safety issue at Tesla is that of being swatted by your CEO for pointing out that resin is being injected into the holes of unsafe lithium batteries and shipped into a bunch of customers' cars

some friends of mine worked with the guy, the jury is still out on that one. ol' musky isn't totally paranoid - we did catch bad actors doing stuff and they were nailed to the wall. finding a real apt in your network can be some next level poo poo

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


the firmware repo was that size if you take into account a huge company, many devices in the car at play and incremental updates to firmware across all those devices + branches for people to do work in. i contributed to that mess by policy, not by choice, but whatever. i'd imagine they'd be smart enough to move to something like git lfs so it isn't as much of a pain

scale stuff:

tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks

i had to deal with marketing people sincerely asking me why we weren't going to run containers on the car in firmware. no, marketing, i don't care that the car would "update faster" or "features would release faster"

a web front-end (we'll say it's a cms that's php-based) that needed $500k in WAF bullshit just so we didn't get pwned every 5 minutes

fragmented installs of splunk. i think i counted well over 20 installs for various departments before they finally hired a decent data scientist that cleaned it up

so many random java, django, .net services from various places, more than i could count and i had to touch a _lot_ of them with firmware. ActiveRecord controlling way way _way_ too much. i consider this probably one of tesla's biggest scale problems - i don't think they actually know or can track exactly what they're running server side at all - so you end up with teams running vmware, nsx, k8s, openstack, hyper-v.

a car that has a json parser implemented in bash 3 because <interpreted language> is dangerous in the car. there are some seriously magic shell scripts on that thing that probably 3 people in the company understand in full

nodejs was a thing for a while but quickly broke down once we reached the 20k car mark - ended up replacing a bunch of that stuff with a Go variant

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


GWBBQ posted:

bets on whether the fire was due to incompetence, act of nature, or deliberately set?


never attribute to malice what can more easily be explained by incompetence

GWBBQ posted:

not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on defective bottles and valves. do you think that attitude might have scared them into not saying anything?

absolutely. taking advantage of the "open door policy" was the fastest way to lose your job at tesla and from what i'm told, spacex, being run by the same guy was no different. there is so much pressure to ship on time they push people to work 14 hour days, 7 days a week - i did that for a while before i just couldn't take it anymore and just accepted being marked down in employee review for being late

Adbot
ADBOT LOVES YOU

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end?

imagine if all of your services were synchronous instead of async

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«2 »