Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
«513 »
  • Post
  • Reply
Cocoa Crispies
Jul 20, 2001

Vehicular Manslaughter!



Pillbug

Xealous posted:

* firmware git repo so large they had to mirror it (something like 2TB)

hardware/firmware people love to commit big binary files to git, 100% plausible

Adbot
ADBOT LOVES YOU

Fatty Crabcakes
Jan 31, 2008

You are not my supervisor.

MononcQc posted:

This thread is a blessing and now one of the best things to hit these forums in the last few years
Definitely Some Quality Poasts℠

And not just because I like reading about shitshows I am not responsible for.

Rex-Goliath
Nov 12, 2016



Cocoa Crispies posted:

hardware/firmware people love to commit big binary files to git, 100% plausible

i’ve been yelling at some java devs to stop loving checking their jars into svn i 100% buy firmware devs doing the same thing but worse

Fatty Crabcakes
Jan 31, 2008

You are not my supervisor.

Rex-Goliath posted:

i’ve been yelling at some java devs to stop loving checking their jars into svn i 100% buy firmware devs doing the same thing but worse
At one place we instituted a policy of pranking anyone who did this.

Mozzie
Oct 26, 2007


I fuckin' re-logged in after nearly a decade to make this single post.



lol

qirex
Feb 15, 2001



we’ve reached the “insurance fraud” stage of the scam I see

Dumb Lowtax
Jul 9, 2005

Save blue hippo from danger, vote for Moo Moo in the best cat contest

Plaster Town Cop

Mozzie posted:

I fuckin' re-logged in after nearly a decade to make this single post.



lol

creationist believer
Feb 16, 2007



College Slice

ty elon and op for this yosrenaissance

Farmer Crack-Ass
Jan 2, 2001

Poyo!


Xealous posted:

just remembered some bits of trivia

* it networking so bad the company had permanent 5~8% consistent packet loss between various places (like, next rack)


holy poo poo

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


the firmware repo was that size if you take into account a huge company, many devices in the car at play and incremental updates to firmware across all those devices + branches for people to do work in. i contributed to that mess by policy, not by choice, but whatever. i'd imagine they'd be smart enough to move to something like git lfs so it isn't as much of a pain

scale stuff:

tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks

i had to deal with marketing people sincerely asking me why we weren't going to run containers on the car in firmware. no, marketing, i don't care that the car would "update faster" or "features would release faster"

a web front-end (we'll say it's a cms that's php-based) that needed $500k in WAF bullshit just so we didn't get pwned every 5 minutes

fragmented installs of splunk. i think i counted well over 20 installs for various departments before they finally hired a decent data scientist that cleaned it up

so many random java, django, .net services from various places, more than i could count and i had to touch a _lot_ of them with firmware. ActiveRecord controlling way way _way_ too much. i consider this probably one of tesla's biggest scale problems - i don't think they actually know or can track exactly what they're running server side at all - so you end up with teams running vmware, nsx, k8s, openstack, hyper-v.

a car that has a json parser implemented in bash 3 because <interpreted language> is dangerous in the car. there are some seriously magic shell scripts on that thing that probably 3 people in the company understand in full

nodejs was a thing for a while but quickly broke down once we reached the 20k car mark - ended up replacing a bunch of that stuff with a Go variant

GWBBQ
Jan 2, 2005



bets on whether the fire was due to incompetence, act of nature, or deliberately set?

Xealous posted:

he's never wrong. his "open door policy" was an invitation to catch you breaking rank.
not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on defective bottles and valves. do you think that attitude might have scared them into not saying anything?

Fuzzy Mammal
Aug 15, 2001



Lipstick Apathy

lmao at least the sales side is going we-


quote:

This “owner adviser” says he left a job paying $150,000 a year for a Tesla base salary of $34,000, with Tesla execs promising enough in annual commissions to match his previous salary. He soon learned that would never happen, because Tesla keeps moving the goal posts.

“I had a friend who killed it — her commission check was going to be $42,000,” he says. “They said, ‘Just kidding. You missed [your goal]. It’s going to be $4,000 for the year.’”

TerminalRaptor
Nov 6, 2012

Mostly Harmless


was there any sort of accountability for the devs there, or was it if you knew how to talk the talk you could bs your way through the ranks while producing nothing of value?


was there any noticeable increase in the absurdity of musk's requests as time went on? anything particularly absurd he called for that was flat out shot down?

TerminalRaptor fucked around with this message at Aug 24, 2018 around 04:57

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


Xealous posted:

tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks

i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end?

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


GWBBQ posted:

bets on whether the fire was due to incompetence, act of nature, or deliberately set?


never attribute to malice what can more easily be explained by incompetence

GWBBQ posted:

not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on defective bottles and valves. do you think that attitude might have scared them into not saying anything?

absolutely. taking advantage of the "open door policy" was the fastest way to lose your job at tesla and from what i'm told, spacex, being run by the same guy was no different. there is so much pressure to ship on time they push people to work 14 hour days, 7 days a week - i did that for a while before i just couldn't take it anymore and just accepted being marked down in employee review for being late

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end?

imagine if all of your services were synchronous instead of async

Farmer Crack-Ass
Jan 2, 2001

Poyo!


infernal machines posted:

i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end?

guessing it's related to these nuggets here:


Xealous posted:

model s and x use openvpn to talk to their backend. inside that backend there are metadata services that feed info to the system, one of those things being a ~20MB+ (generated by the worst erp system) json payload that describes supercharger poo poo for the map in the touchscreen.

Xealous posted:

more fun facts:

the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again.

Xealous posted:

more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.

eschaton
Mar 7, 2007
Ask me how it all ends!

Oven Wrangler

all sounding like when Musk fired Lattner (I presume because he was realistic about the current state of what he was responsible for, what could be achieved in those areas, and the timeframes in which that could happen) the board should have just fired Musk instead and maybe made Lattner his replacement

(Chris knows how to hire good people and listen to them)

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


the openvpn problem is easy to get around thundering herd/scale issues if you design it correctly and know how to run a network. in theory, you could get around a lot openvpn scale issues if you use bridged networking, ipv6 on the inside, and some redundant dhcp servers to hand out leases - that kind of poo poo won't work in most cloud providers though so you stuck at running that crap in a datacenter.

tesla's issues around the services were many fold - the specifics would give away too much, but i'll say this: when you make all of your services depend on a single rdbms while simultaneously using the world's worst ORM, you get what's coming to you.

i poked around on a 3 a friend has and after looking at a packet cap it looks like they're doing ssl'd amqp - i didn't see any openvpn packets so i suspect they got wise to how lovely it can be, but lol at running connected car stuff directly over the internet outside a private apn or a tunnel

hobbesmaster
Jan 28, 2008


wait they’re not using a private apn?!

FAUXTON
Jun 2, 2005

daef


thundering herd problem more like thundering turd problem

The staggering level of internal fragmentation reminds me of how PayPal was when I worked there in '09-15. They experimented for a few months with an "agile product solutions" team that basically took "we need a widget that does this" orders and cranked out custom Java poo poo that never worked.

We were cataloguing suspect addresses in an access database. They'd eventually gotten so wrapped up in quantity over quality that we were amending filings that weren't even sent to fincen yet, there'd be multiple SAR filings in the batch that were a daisy chain of amendments and corrections. Good god I'd love to run across someone from the treasury who had to deal with that poo poo and apologize.

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


hobbesmaster posted:

wait they’re not using a private apn?!

can't say - i do most of the packet grabbin' on wifi because that thing will shakes its bits on any ole network

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


FAUXTON posted:

The staggering level of internal fragmentation reminds me of how PayPal was when I worked there in '09-15. They experimented for a few months with an "agile product solutions" team that basically took "we need a widget that does this" orders and cranked out custom Java poo poo that never worked.

that's basically tesla in a nutshell only, i guess it kinda works. every different team has some kind of different service where you can get data but none of it published anywhere, there are no standards, and everyone just loves to write their own client implementations because they don't trust you to do it right (sorry that we don't have a client in C++ which is mandated by policy for the car)

poking holes in the firewall was always super fun - i would describe, in full detail all ports, sources, destinations, have security assessments done, etc and somehow, still, the firewall cj's would gently caress up the ports. i once spent, and this is not a joke, 3 weeks chasing a single port down - i think that email thread had 100 reply-all's, two video confs and me visiting the firewall cj in fremont before it was finally fixed

fart simpson
Jul 2, 2005



Lipstick Apathy

Xealous posted:

that's basically tesla in a nutshell only, i guess it kinda works. every different team has some kind of different service where you can get data but none of it published anywhere, there are no standards, and everyone just loves to write their own client implementations because they don't trust you to do it right (sorry that we don't have a client in C++ which is mandated by policy for the car)

poking holes in the firewall was always super fun - i would describe, in full detail all ports, sources, destinations, have security assessments done, etc and somehow, still, the firewall cj's would gently caress up the ports. i once spent, and this is not a joke, 3 weeks chasing a single port down - i think that email thread had 100 reply-all's, two video confs and me visiting the firewall cj in fremont before it was finally fixed

good poo poo op

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


TerminalRaptor posted:

was there any sort of accountability for the devs there, or was it if you knew how to talk the talk you could bs your way through the ranks while producing nothing of value?


was there any noticeable increase in the absurdity of musk's requests as time went on? anything particularly absurd he called for that was flat out shot down?

no, if you didn't do work it was really really obvious and they purged you quickly. that didn't mean it was any _good_ but if you produced you were generally left to your own devices as long as you weren't breaking builds - this seemed to be true of most engineering teams.

ol' musky did increasingly weird poo poo, but i wouldn't necessarily call it out of the ordinary for silicon valley - many folks, me included, for a time, viewed him as a bit of a Jobs-type. his behavior became really erratic around the time we wrapped up X and headed for 3 full steam - the more stuff piling on about autopilot, the more issues with the factory, the ongoing issues with X and then with 3 mfg, his ongoing spacex work - the dude really needs a nap and to just walk away from tesla at this point. its arguable he isn't running it successfully considering all the issues

* edit - running it successfully by silicon valley standards. too many issues to reach profitability because of really poor strategy and execution. too many people get wrapped up in his celebrity without really asking 'can he pull this off' which is the difference between him and Jobs - Jobs actually did poo poo

fart-powered cars fucked around with this message at Aug 24, 2018 around 05:59

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


is turning a profit considered an issue?

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

is turning a profit considered an issue?

consider silicon valley when you ask that question

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


yeah, i get that, it's just they make a product that will probably poo poo itself when the back end goes dark, and that product costs $65k-$120k so it's an outlier by sv standards.

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


Xealous posted:

* edit - running it successfully by silicon valley standards. too many issues to reach profitability because of really poor strategy and execution. too many people get wrapped up in his celebrity without really asking 'can he pull this off' which is the difference between him and Jobs - Jobs actually did poo poo

jobs got booted from his own company for a decade and his only saving grace was that a soft drink exec managed to misunderstand the market even worse than he did. his non-apple 90s projects were commercial failures

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

yeah, i get that, it's just they make a product that will probably poo poo itself when the back end goes dark, and that product costs $65k-$120k so it's an outlier by sv standards.

the product shouldn't poo poo itself when the backend eventually goes dark - autopilot won't work, updates won't, remote phone poo poo won't but otherwise the driving and infotainment part of the car should still function if you pull the sim and put your own in. given how poo poo the firmware security is it'd be pretty easy to dump the firmware, compile up some statically linked tools for shits and just patch in your own services. there's been a few clever people on twitter who figured out you can run Go arm bins on the thing - after that it's just figuring out what crap you care about on CAN (if anything).

all that said, tesla did sell cars explicitly with the sim pulled and no network ever - service was always complaining to us because the ring logs on those cars would take hours to parse.

speaking of the ring logs - because there was no battery backed rtc, we had to stitch and best-guess times based on the intervals when the car _did_ have valid time and patch that into the logs serially before they could be imported. inaccuracies in the signal data could and did lead to all kinds of bullshit when somebody needed to be debug issues

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

jobs got booted from his own company for a decade and his only saving grace was that a soft drink exec managed to misunderstand the market even worse than he did. his non-apple 90s projects were commercial failures

yup - i suspect elon is more like 90's era jobs - but i don't think it's a market understanding problem. tesla has really good engineering on the go-juice part of the car, but really really lacks in total execution: fit, finish, body, interior, design. if nothing else they could be profitable selling battery packs, drive trains, their new in-house chips, that kind of thing.

Roosevelt
Jul 18, 2009

I'm looking for the man who shot my paw.


jesus christ what a mess

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


y'all were miles ahead on relatively cheap batteries and thermal/charge management so they don't poo poo themselves after a year of use. unfortunately, that advantage doen't last forever

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


Xealous posted:

all that said, tesla did sell cars explicitly with the sim pulled and no network ever - service was always complaining to us because the ring logs on those cars would take hours to parse.

also, what? can you expand on why/what markets?

Roosevelt
Jul 18, 2009

I'm looking for the man who shot my paw.


elon musk is a dumb arrogant fuckin bastard

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

also, what? can you expand on why/what markets?

can't expand on anything customer/market related. it is an option during the manufacturing process, you just have to ask for it. as for the why: you can't remotely root, monitor or otherwise track a 'connected car' that isn't actually connected

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


i'm surprised that's an option

fart-powered cars
Apr 19, 2001

I WENT VIRAL AND ALL I GOT WAS THIS LOUSY AVATAR


infernal machines posted:

i'm surprised that's an option

it isn't advertised for obvious reasons. tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours. pii was stripped, but i don't think they could have pulled it off quite the same way if the US had GDPR.

fart simpson
Jul 2, 2005



Lipstick Apathy

Xealous posted:

it isn't advertised for obvious reasons. tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours. pii was stripped, but i don't think they could have pulled it off quite the same way if the US had GDPR.

yeah gdpr owns

Adbot
ADBOT LOVES YOU

infernal machines
Oct 11, 2012

Don't let the little fuckers generation gap you


Xealous posted:

tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours.

this raises so many questions i don't even know where to start. not even on the privacy front, just how that data could be useful for autonomous lane keeping and cruise control

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«513 »