|
Xealous posted:* firmware git repo so large they had to mirror it (something like 2TB) hardware/firmware people love to commit big binary files to git, 100% plausible
|
#
?
Aug 24, 2018 03:34
|
|
|
|
| # ? Apr 16, 2024 09:19 |
|
|
MononcQc posted:This thread is a blessing and now one of the best things to hit these forums in the last few years And not just because I like reading about shitshows I am not responsible for.
|
|
#
?
Aug 24, 2018 03:36
|
|
|
Cocoa Crispies posted:hardware/firmware people love to commit big binary files to git, 100% plausible i’ve been yelling at some java devs to stop loving checking their jars into svn i 100% buy firmware devs doing the same thing but worse
|
|
#
?
Aug 24, 2018 03:53
|
|
|
Rex-Goliath posted:i’ve been yelling at some java devs to stop loving checking their jars into svn i 100% buy firmware devs doing the same thing but worse
|
|
#
?
Aug 24, 2018 04:10
|
|
|
I fuckin' re-logged in after nearly a decade to make this single post. lol
|
|
#
?
Aug 24, 2018 04:35
|
|
|
we’ve reached the “insurance fraud” stage of the scam I see
|
|
#
?
Aug 24, 2018 04:47
|
|
|
Mozzie posted:I fuckin' re-logged in after nearly a decade to make this single post.
|
|
#
?
Aug 24, 2018 05:15
|
|
|
ty elon and op for this yosrenaissance
|
|
#
?
Aug 24, 2018 05:23
|
|
|
Xealous posted:just remembered some bits of trivia holy poo poo
|
|
#
?
Aug 24, 2018 05:40
|
|
|
the firmware repo was that size if you take into account a huge company, many devices in the car at play and incremental updates to firmware across all those devices + branches for people to do work in. i contributed to that mess by policy, not by choice, but whatever. i'd imagine they'd be smart enough to move to something like git lfs so it isn't as much of a pain scale stuff: tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks i had to deal with marketing people sincerely asking me why we weren't going to run containers on the car in firmware. no, marketing, i don't care that the car would "update faster" or "features would release faster"  a web front-end (we'll say it's a cms that's php-based) that needed $500k in WAF bullshit just so we didn't get pwned every 5 minutes fragmented installs of splunk. i think i counted well over 20 installs for various departments before they finally hired a decent data scientist that cleaned it up so many random java, django, .net services from various places, more than i could count and i had to touch a _lot_ of them with firmware. ActiveRecord controlling way way _way_ too much. i consider this probably one of tesla's biggest scale problems - i don't think they actually know or can track exactly what they're running server side at all - so you end up with teams running vmware, nsx, k8s, openstack, hyper-v. a car that has a json parser implemented in bash 3 because <interpreted language> is dangerous in the car. there are some seriously magic shell scripts on that thing that probably 3 people in the company understand in full nodejs was a thing for a while but quickly broke down once we reached the 20k car mark - ended up replacing a bunch of that stuff with a Go variant
|
|
#
?
Aug 24, 2018 05:41
|
|
|
bets on whether the fire was due to incompetence, act of nature, or deliberately set?Xealous posted:he's never wrong. his "open door policy" was an invitation to catch you breaking rank.
|
|
#
?
Aug 24, 2018 05:52
|
|
|
lmao at least the sales side is going we-quote:This “owner adviser” says he left a job paying $150,000 a year for a Tesla base salary of $34,000, with Tesla execs promising enough in annual commissions to match his previous salary. He soon learned that would never happen, because Tesla keeps moving the goal posts.
|
|
#
?
Aug 24, 2018 05:52
|
|
|
was there any sort of accountability for the devs there, or was it if you knew how to talk the talk you could bs your way through the ranks while producing nothing of value? was there any noticeable increase in the absurdity of musk's requests as time went on? anything particularly absurd he called for that was flat out shot down? TerminalRaptor fucked around with this message at 05:57 on Aug 24, 2018 |
|
#
?
Aug 24, 2018 05:54
|
|
|
Xealous posted:tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production poo poo come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peaks i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end?
|
|
#
?
Aug 24, 2018 05:56
|
|
|
GWBBQ posted:bets on whether the fire was due to incompetence, act of nature, or deliberately set? never attribute to malice what can more easily be explained by incompetence GWBBQ posted:not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on defective bottles and valves. do you think that attitude might have scared them into not saying anything? absolutely. taking advantage of the "open door policy" was the fastest way to lose your job at tesla and from what i'm told, spacex, being run by the same guy was no different. there is so much pressure to ship on time they push people to work 14 hour days, 7 days a week - i did that for a while before i just couldn't take it anymore and just accepted being marked down in employee review for being late
|
|
#
?
Aug 24, 2018 05:57
|
|
|
infernal machines posted:i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end? imagine if all of your services were synchronous instead of async
|
|
#
?
Aug 24, 2018 05:59
|
|
|
infernal machines posted:i realize you may not be able to answer this for reasons, but what the christ are your vehicles doing that ends up with this much traffic on your back end? guessing it's related to these nuggets here: Xealous posted:model s and x use openvpn to talk to their backend. inside that backend there are metadata services that feed info to the system, one of those things being a ~20MB+ (generated by the worst erp system) json payload that describes supercharger poo poo for the map in the touchscreen. Xealous posted:more fun facts: Xealous posted:more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.
|
|
#
?
Aug 24, 2018 05:59
|
|
|
all sounding like when Musk fired Lattner (I presume because he was realistic about the current state of what he was responsible for, what could be achieved in those areas, and the timeframes in which that could happen) the board should have just fired Musk instead and maybe made Lattner his replacement (Chris knows how to hire good people and listen to them)
|
|
#
?
Aug 24, 2018 05:59
|
|
|
the openvpn problem is easy to get around thundering herd/scale issues if you design it correctly and know how to run a network. in theory, you could get around a lot openvpn scale issues if you use bridged networking, ipv6 on the inside, and some redundant dhcp servers to hand out leases - that kind of poo poo won't work in most cloud providers though so you stuck at running that crap in a datacenter. tesla's issues around the services were many fold - the specifics would give away too much, but i'll say this: when you make all of your services depend on a single rdbms while simultaneously using the world's worst ORM, you get what's coming to you. i poked around on a 3 a friend has and after looking at a packet cap it looks like they're doing ssl'd amqp - i didn't see any openvpn packets so i suspect they got wise to how lovely it can be, but lol at running connected car stuff directly over the internet outside a private apn or a tunnel
|
|
#
?
Aug 24, 2018 06:10
|
|
|
wait they’re not using a private apn?!
|
|
#
?
Aug 24, 2018 06:12
|
|
|
thundering herd problem more like thundering turd problem The staggering level of internal fragmentation reminds me of how PayPal was when I worked there in '09-15. They experimented for a few months with an "agile product solutions" team that basically took "we need a widget that does this" orders and cranked out custom Java poo poo that never worked. We were cataloguing suspect addresses in an access database. They'd eventually gotten so wrapped up in quantity over quality that we were amending filings that weren't even sent to fincen yet, there'd be multiple SAR filings in the batch that were a daisy chain of amendments and corrections. Good god I'd love to run across someone from the treasury who had to deal with that poo poo and apologize.
|
|
#
?
Aug 24, 2018 06:14
|
|
|
hobbesmaster posted:wait they’re not using a private apn?! can't say - i do most of the packet grabbin' on wifi because that thing will shakes its bits on any ole network
|
|
#
?
Aug 24, 2018 06:15
|
|
|
FAUXTON posted:The staggering level of internal fragmentation reminds me of how PayPal was when I worked there in '09-15. They experimented for a few months with an "agile product solutions" team that basically took "we need a widget that does this" orders and cranked out custom Java poo poo that never worked. that's basically tesla in a nutshell only, i guess it kinda works. every different team has some kind of different service where you can get data but none of it published anywhere, there are no standards, and everyone just loves to write their own client implementations because they don't trust you to do it right (sorry that we don't have a client in C++ which is mandated by policy for the car) poking holes in the firewall was always super fun - i would describe, in full detail all ports, sources, destinations, have security assessments done, etc and somehow, still, the firewall cj's would gently caress up the ports. i once spent, and this is not a joke, 3 weeks chasing a single port down - i think that email thread had 100 reply-all's, two video confs and me visiting the firewall cj in fremont before it was finally fixed
|
|
#
?
Aug 24, 2018 06:26
|
|
|
Xealous posted:that's basically tesla in a nutshell only, i guess it kinda works. every different team has some kind of different service where you can get data but none of it published anywhere, there are no standards, and everyone just loves to write their own client implementations because they don't trust you to do it right (sorry that we don't have a client in C++ which is mandated by policy for the car) good poo poo op
|
|
#
?
Aug 24, 2018 06:28
|
|
|
TerminalRaptor posted:was there any sort of accountability for the devs there, or was it if you knew how to talk the talk you could bs your way through the ranks while producing nothing of value? no, if you didn't do work it was really really obvious and they purged you quickly. that didn't mean it was any _good_ but if you produced you were generally left to your own devices as long as you weren't breaking builds - this seemed to be true of most engineering teams. ol' musky did increasingly weird poo poo, but i wouldn't necessarily call it out of the ordinary for silicon valley - many folks, me included, for a time, viewed him as a bit of a Jobs-type. his behavior became really erratic around the time we wrapped up X and headed for 3 full steam - the more stuff piling on about autopilot, the more issues with the factory, the ongoing issues with X and then with 3 mfg, his ongoing spacex work - the dude really needs a nap and to just walk away from tesla at this point. its arguable he isn't running it successfully considering all the issues * edit - running it successfully by silicon valley standards. too many issues to reach profitability because of really poor strategy and execution. too many people get wrapped up in his celebrity without really asking 'can he pull this off' which is the difference between him and Jobs - Jobs actually did poo poo fart-powered cars fucked around with this message at 06:59 on Aug 24, 2018 |
|
#
?
Aug 24, 2018 06:50
|
|
|
is turning a profit considered an issue?
|
|
#
?
Aug 24, 2018 06:53
|
|
|
infernal machines posted:is turning a profit considered an issue? consider silicon valley when you ask that question
|
|
#
?
Aug 24, 2018 06:54
|
|
|
yeah, i get that, it's just they make a product that will probably poo poo itself when the back end goes dark, and that product costs $65k-$120k so it's an outlier by sv standards.
|
|
#
?
Aug 24, 2018 07:10
|
|
|
Xealous posted:* edit - running it successfully by silicon valley standards. too many issues to reach profitability because of really poor strategy and execution. too many people get wrapped up in his celebrity without really asking 'can he pull this off' which is the difference between him and Jobs - Jobs actually did poo poo jobs got booted from his own company for a decade and his only saving grace was that a soft drink exec managed to misunderstand the market even worse than he did. his non-apple 90s projects were commercial failures
|
|
#
?
Aug 24, 2018 07:18
|
|
|
infernal machines posted:yeah, i get that, it's just they make a product that will probably poo poo itself when the back end goes dark, and that product costs $65k-$120k so it's an outlier by sv standards. the product shouldn't poo poo itself when the backend eventually goes dark - autopilot won't work, updates won't, remote phone poo poo won't but otherwise the driving and infotainment part of the car should still function if you pull the sim and put your own in. given how poo poo the firmware security is it'd be pretty easy to dump the firmware, compile up some statically linked tools for shits and just patch in your own services. there's been a few clever people on twitter who figured out you can run Go arm bins on the thing - after that it's just figuring out what crap you care about on CAN (if anything). all that said, tesla did sell cars explicitly with the sim pulled and no network ever - service was always complaining to us because the ring logs on those cars would take hours to parse. speaking of the ring logs - because there was no battery backed rtc, we had to stitch and best-guess times based on the intervals when the car _did_ have valid time and patch that into the logs serially before they could be imported. inaccuracies in the signal data could and did lead to all kinds of bullshit when somebody needed to be debug issues
|
|
#
?
Aug 24, 2018 07:38
|
|
|
infernal machines posted:jobs got booted from his own company for a decade and his only saving grace was that a soft drink exec managed to misunderstand the market even worse than he did. his non-apple 90s projects were commercial failures yup - i suspect elon is more like 90's era jobs - but i don't think it's a market understanding problem. tesla has really good engineering on the go-juice part of the car, but really really lacks in total execution: fit, finish, body, interior, design. if nothing else they could be profitable selling battery packs, drive trains, their new in-house chips, that kind of thing.
|
|
#
?
Aug 24, 2018 07:43
|
|
|
jesus christ what a mess
|
|
#
?
Aug 24, 2018 07:44
|
|
|
y'all were miles ahead on relatively cheap batteries and thermal/charge management so they don't poo poo themselves after a year of use. unfortunately, that advantage doen't last forever
|
|
#
?
Aug 24, 2018 07:47
|
|
|
Xealous posted:all that said, tesla did sell cars explicitly with the sim pulled and no network ever - service was always complaining to us because the ring logs on those cars would take hours to parse. also, what? can you expand on why/what markets?
|
|
#
?
Aug 24, 2018 07:48
|
|
|
elon musk is a dumb arrogant fuckin bastard
|
|
#
?
Aug 24, 2018 07:51
|
|
|
infernal machines posted:also, what? can you expand on why/what markets? can't expand on anything customer/market related. it is an option during the manufacturing process, you just have to ask for it. as for the why: you can't remotely root, monitor or otherwise track a 'connected car' that isn't actually connected
|
|
#
?
Aug 24, 2018 07:52
|
|
|
i'm surprised that's an option
|
|
#
?
Aug 24, 2018 07:58
|
|
|
infernal machines posted:i'm surprised that's an option it isn't advertised for obvious reasons. tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours. pii was stripped, but i don't think they could have pulled it off quite the same way if the US had GDPR.
|
|
#
?
Aug 24, 2018 08:01
|
|
|
Xealous posted:it isn't advertised for obvious reasons. tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours. pii was stripped, but i don't think they could have pulled it off quite the same way if the US had GDPR. yeah gdpr owns
|
|
#
?
Aug 24, 2018 08:08
|
|
|
|
| # ? Apr 16, 2024 09:19 |
|
|
Xealous posted:tesla was partially able to bring autopilot to market so quickly because they had seeded all cars after a certain time with autopilot sensors and hardware, just not activated - they then dialed up the tracking across the customer fleet to gather the data - and with the density of cars they had at the time with hardware in, they had a goldmine within hours. this raises so many questions i don't even know where to start. not even on the privacy front, just how that data could be useful for autonomous lane keeping and cruise control
|
|
#
?
Aug 24, 2018 08:16
|
|