Ruby on Rails Love-In

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ruby on Rails Love-In

«‹›178 »

Doh004: Apr 22, 2007; Mmmmm Donuts...

Come on over to the normalized data land:

Vehicles
- id
- vehicle_type_id
- number_of_wheels

VehicleType
- id
- name (human readable)

VehiclePropertyTypes
- id
- property_name
- property_value_type (could help you cast your values later on, not necessary)

VehicleTypeProprerties
- vehicle_type_id
- vehicle_property_type_id

VehicleProperties
- vehicle_id
- vehicle_property_type_id
- value

Joins are your friend, my friend :getin:

# ? Mar 7, 2018 04:31

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 20:25

Peristalsis: Apr 5, 2004; Move along.

Thanks for your STI input. It sounds like STI is sort of there because it's easy and often good enough.

Now I have a question about FactoryBot best practices. Using sequences for unique names is fine with RSpec tests, but if you use a factory to generate dev data from Rails console, the sequences restart every time you restart the console, which means they aren't unique any more. I've been replacing sequence and Faker::Name.unique instances with names with timestamps appended to manually unique-ify things as collisions come up. Another dev doesn't feel that FactoryBot should be used in Rails console in this way, and that it's only for setting up and executing tests. I think it's crazy to manually set up data with multiple complex associations by hand every time I want to try something out when that's exactly what factories do for us. Is he right that it's a best practice not to use Factories from the console? We also use factories in our seeds.rb for populating dev data - I'm not sure why that would be okay, but creating more data from the console isn't. (And I just found out that we may have a factory being used in some production data import code, too - I'm pretty sure that isn't a good idea.)

# ? Mar 7, 2018 17:19

necrotic: Aug 2, 2005; I owe my brother big time for this!

It's fine to use factory bot for dev data. Almost definitely a bad idea for prod data. If that dev isn't against the seed usage he's being dense.

# ? Mar 7, 2018 17:25

Slimy Hog: Apr 22, 2008

I'm not even sure how you would use a factory in production.

# ? Mar 7, 2018 19:56

Peristalsis: Apr 5, 2004; Move along.

Slimy Hog posted:

I'm not even sure how you would use a factory in production.

I haven't verified that it's there (another developer told me), but it's for a data import from another system. I assume it's just a way to quickly create model objects from values read in from a text file.

Edit: Yeah, its used extensively in an import rake task.

Peristalsis fucked around with this message at 20:10 on Mar 7, 2018

# ? Mar 7, 2018 20:08

KoRMaK: Jul 31, 2012

We have one STI thing in our app, I inherited it (meaning someone else implemented it before I got on the project) and really I wouldn't do it again. It hasn't really saved me too much time, and it just complicates things. That part of the app always needs extra attention and QA because it's different than the rest of the app.

Then we added the ability to add custom fields to any object and things get real crazy when users try to export/import :unsmigghh:

# ? Mar 7, 2018 20:44

xtal: Jan 9, 2011; by Fluffdaddy

Peristalsis posted:

I haven't verified that it's there (another developer told me), but it's for a data import from another system. I assume it's just a way to quickly create model objects from values read in from a text file.

Edit: Yeah, its used extensively in an import rake task.

I think one of us is misunderstanding factories

# ? Mar 7, 2018 23:27

Peristalsis: Apr 5, 2004; Move along.

xtal posted:

I think one of us is misunderstanding factories

That's possible. I'm talking about the FactoryBot gem used to set up test data, not the standard factory design pattern. I'm also very tired today, so I apologize if I garbled my descriptions.

# ? Mar 8, 2018 00:28

GraceGarland: Jul 4, 2003

Peristalsis posted:

I'm a little late to the STI discussion, but is there an accepted better way to model inheritance of Model classes in Rails? The only other way I've seen is to have a separate table for each child model, with references back to the parent model. So, something like this:

Polymorphic ~~inheritance~~ associations maybe?

GraceGarland fucked around with this message at 17:12 on Mar 8, 2018

# ? Mar 8, 2018 07:47

Peristalsis: Apr 5, 2004; Move along.

ToadStyle posted:

Polymorphic inheritance maybe?

I'm not sure what you mean. Isn't all inheritance polymorphic by definition? I'm specifically asking about how to model class hierarchies in relational database tables, not how to structure them in code.

# ? Mar 8, 2018 16:21

GraceGarland: Jul 4, 2003

Peristalsis posted:

I'm not sure what you mean. Isn't all inheritance polymorphic by definition? I'm specifically asking about how to model class hierarchies in relational database tables, not how to structure them in code.

Sorry, misspoke. "Polymorphic Associations" was what I meant, but I see you actually mentioned that in the text I quoted. Just ignore me.

# ? Mar 8, 2018 17:12

Pollyanna: Mar 5, 2005; Milk's on them.

�nother ActiveRecord question. I have a SQL query I want to translate into ActiveRecord calls. I want to write a where clause that compares columns in two different tables on a join clause, and then reference both tables in a block, like so:

code:

User.group(:id).joins(:entries).where(�entries.created_at < users.joined_at�).each do |user|
  first_entry_date = user.entries.order(:created_at).first.created_at
  user.joined_at = first_entry_date
end

Is this the most efficient way to write the code? It doesn�t get the exact SQL clause I wanted, but it seems to do the job. How do I ensure that it joins and groups by the earliest entry by created_at in the initial join? Is there a tool/method that would do this for me and I just don�t know about it?

# ? Mar 27, 2018 14:14

xtal: Jan 9, 2011; by Fluffdaddy

Pollyanna posted:

�nother ActiveRecord question. I have a SQL query I want to translate into ActiveRecord calls. I want to write a where clause that compares columns in two different tables on a join clause, and then reference both tables in a block, like so:
code:
User.group(:id).joins(:entries).where(�entries.created_at < users.joined_at�).each do |user|
  first_entry_date = user.entries.order(:created_at).first.created_at
  user.joined_at = first_entry_date
end 
Is this the most efficient way to write the code? It doesn�t get the exact SQL clause I wanted, but it seems to do the job. How do I ensure that it joins and groups by the earliest entry by created_at in the initial join? Is there a tool/method that would do this for me and I just don�t know about it?

I'm too tired to phrase this in a defensive way, so I will say what I think and someone can tell me if I'm a dumb idiot moron.

1. That's an n+1 query because you're doing one call to load the users, then one call for each user inside the loop. Try to preload that information so it doesn't scale with the number of records.
2. That code can explode in production because it will load every applicable user object into memory at once. Use find_each instead of each for batching.
3. You can probably do this in one query with something like the following that I haven't tested or even executed.

Ruby code:

User.joins(:entries).where('entries.created_at < users.joined_at').update_all('joined_at = min(entries.created_at)')

xtal fucked around with this message at 00:38 on Mar 28, 2018

# ? Mar 28, 2018 00:36

A MIRACLE: Sep 17, 2007; All right. It's Saturday night; I have no date, a two-liter bottle of Shasta and my all-Rush mix-tape... Let's rock.

Post your raw sql that�s working and I�ll try to come up with a better query

Also you don�t have to do everything with ActiveRecord aliases, sometimes it�s easier and makes sense to use raw sql

# ? Mar 28, 2018 00:46

Pollyanna: Mar 5, 2005; Milk's on them.

Good points on the efficiency of the code. My raw SQL is something like

code:

select
  u.id,
  u.joined_at,
  min(e.created_at) as first_entry_created_at
from users u
join entries e on e.user_id = u.id
where e.created_at < u.joined_at
group by 1;

This is just for selecting them, actual updating I�m not sure what to do. I was gonna go through ActiveRecord for that.

# ? Mar 28, 2018 15:27

necrotic: Aug 2, 2005; I owe my brother big time for this!

If all you want to do is update that joined at column then xtal's approach should work and is a single query for the whole process. Note that it will bypass validations of your user model, but that likely doesn't matter for this case.

# ? Mar 28, 2018 15:51

Pollyanna: Mar 5, 2005; Milk's on them.

Xtal�s query unfortunately fails by claiming that entries is not in the FROM clause when min(entries.created_at) is referenced :negative:

I would have expected it to get that info from the subselection it uses to determine the ids it�s working with but I guess not.

Edit: find_each hangs for some reason :/

Pollyanna fucked around with this message at 20:35 on Mar 28, 2018

# ? Mar 28, 2018 19:57

Pollyanna: Mar 5, 2005; Milk's on them.

I fixed the previous issue - we're good!

We're now currently thinking of how to remove MongoDB from the project. We have a main UI project, and an API project that was factored out a little while ago. The UI processes some data from its DB, builds some Mongo documents from them, puts them into Mongo, and the API consumes it. We now want to remove Mongo due to performance concerns and general irritation at the thing, but from what I and another engineer can see the only way to do this is to either:

1. Have the API point directly to the UI DB and build the data it needs itself, or
2. Have the UI build the relevant data, and POST it to the API through some sort of endpoint to the API's own DB.

In either case, both projects will need the same ActiveRecord models, which means we would have to update both projects if something changes that both of them use. (In fact, that's actually currently the case anyway - they often have to be in sync.)

We're considering factoring out the main logic of the application into a gem or something - has anyone done that before? Is it a good idea? Is any of this a good idea?

# ? Apr 11, 2018 23:45

xtal: Jan 9, 2011; by Fluffdaddy

I don't understand what's going on, but it sounds weird. Can you explain more about what these apps do and how they relate? I understand having a UI and API app but I don't get what MongoDB is doing or what you're trying to do instead.

xtal fucked around with this message at 00:40 on Apr 12, 2018

# ? Apr 12, 2018 00:37

Pollyanna: Mar 5, 2005; Milk's on them.

It's a codebase I'm still wrapping my head around, but the tl;dr that I am aware of is that in the interest of moving towards microservices, a large monolithic Rails app had some API endpoints broken out into their own application. This API application was originally going to be in the MEAN stack, but they didn't like it and went back to Rails - except for Mongo, which remained shared between the two apps.

The UI periodically fires off some Sidekiq workers that build some Mongo-backed documents from the UI app's models, build a larger Mongo-backed document from the smaller documents, and upload them to the Mongo DB. The API application, when it receives a query for data, pulls from the shared DB and builds an index of API-specific data.

Both the UI and the API have the same Mongo document configuration (unless they don't), so if we were to move to standard ActiveRecord and away from Mongo, both projects would need the same models and schema. Right now, I don't think I see a way around that, which makes me wonder why the API was broken out in the first place. :iiam:

This is all because Mongo has continually been a pain in the rear end and has performance issues (that may or may not be due to the data modeling).

# ? Apr 12, 2018 02:13

xtal: Jan 9, 2011; by Fluffdaddy

You're suggesting it won't work if you try to share the Rails models between several applications. That's true, and it's also what you did by sharing the MongoDB schema. I think that you need to move to a more formalized consensus protocol or message bus. That said, I am a moron.

# ? Apr 12, 2018 03:47

xtal: Jan 9, 2011; by Fluffdaddy

Furthermore, if your issue with MongoDB is performance and not the fact that you're misusing the technology, there are more direct solutions to that.

# ? Apr 12, 2018 03:49

Less Fat Luke: May 23, 2003; Exciting Lemon

If both the UI and backend are still Rails apps (which it sounds like) you can split out the models into a separate gem, Rails engine or even a git submodule. It's kind of annoying but I've worked in environments where the models were shared via a private gem and it's doable.

# ? Apr 12, 2018 03:52

Pollyanna: Mar 5, 2005; Milk's on them.

That�s pretty much what I expected. We are likely to follow the path of least resistance and simply replace mongo with PG and keep the rest of the hairy logic, but I hope we can clean it up too.

# ? Apr 12, 2018 16:28

EVGA Longoria: Dec 25, 2005; Let's go exploring!

Biggest suggestion I can give you is a database should be used by one app only if you can. It�ll simplify your life 300x. It sounds like you could either split the data into 2 databases, or pass the data via an api call. It�ll make your life so much better.

# ? Apr 12, 2018 21:50

xtal: Jan 9, 2011; by Fluffdaddy

EVGA Longoria posted:

Biggest suggestion I can give you is a database should be used by one app only if you can. It�ll simplify your life 300x. It sounds like you could either split the data into 2 databases, or pass the data via an api call. It�ll make your life so much better.

If this is solution understands the problem, I agree with it a lot. You can split the model layer into a gem, but you will run into problems. It's hard enough to deploy database migrations as it is.

# ? Apr 12, 2018 23:01

Less Fat Luke: May 23, 2003; Exciting Lemon

And your apps commit histories will be forever poo poo up with `bumped model gem` messages

# ? Apr 13, 2018 01:04

Pollyanna: Mar 5, 2005; Milk's on them.

Yeah, I figured. If it was up to me, I would expect there to be a �gimme the relevant data� API in the monolith that the service consumes, and the service build poo poo from that, drawing the data border at service-specific data vs. not specific. I�ll bring this up next week.

# ? Apr 13, 2018 20:04

Pollyanna: Mar 5, 2005; Milk's on them.

Yet more rails questions. This time it�s rspec, and our model specs take forever to run. The entire test suite takes 45 minutes when split across 8 CircleCI machines, but running just the model specs has easily taken a few hours on my local machine. The problem is that each individual test doesn�t take too long (3~5s maybe?), but they really add up. It doesn�t look like there�s a single egregiously bad spec, they�re just all a few to a handful of seconds long and there�s a lot of them.

I�m still waiting on it to report to me exactly what the gently caress is going on, but does anyone have any advice for speeding up a whole bunch of mediocre specs all at once, without yet knowing what�s slowing them all down?

# ? May 4, 2018 19:18

Chilled Milk: Jun 22, 2003; No one here is alone,
satellites in every home

Parallel Tests? Requires a little bit of configuration https://github.com/grosser/parallel_tests

# ? May 4, 2018 19:32

Gmaz: Apr 3, 2011; New DLC for Aoe2 is out: Dynasties of India

Pollyanna posted:

Yet more rails questions. This time it�s rspec, and our model specs take forever to run. The entire test suite takes 45 minutes when split across 8 CircleCI machines, but running just the model specs has easily taken a few hours on my local machine. The problem is that each individual test doesn�t take too long (3~5s maybe?), but they really add up. It doesn�t look like there�s a single egregiously bad spec, they�re just all a few to a handful of seconds long and there�s a lot of them.

I�m still waiting on it to report to me exactly what the gently caress is going on, but does anyone have any advice for speeding up a whole bunch of mediocre specs all at once, without yet knowing what�s slowing them all down?

Can you mock dependencies for test scenarios instead of creating them in the DB, can you just build the model itself instead of writing it to the DB? If you need to change column values to prepare for a test do you do it without triggering callbacks? Can you pull logic outside of models into a PORO and then test that PORO?

These seem like some obvious things to speed up your model specs, hard to tell more without additional context. Handful of seconds for a model spec sounds like a lot of time TBH.

# ? May 4, 2018 20:23

necrotic: Aug 2, 2005; I owe my brother big time for this!

Parallel tests will not help a whole lot if he's already spreading it over 8 servers. Your model tests sound more like full on integration tests if they're taking 3-5 seconds each.

See if there's any common setup you can do before the suite instead of before each test
Are you writing/reading from the database when you don't actually need to? Validation tests, for example, don't always need to persist the model to test.
Do you have a lot of before/after hooks in your models? If you do, I'm sorry. You're boned unless you refactor that poo poo out.

I worked at a rails shop a couple years back that had a 20 minute suite but it ran on 15 nodes with 2 runners each. This was mostly because of the 3rd point. Running the whole suite locally took hours.

# ? May 4, 2018 20:27

xtal: Jan 9, 2011; by Fluffdaddy

Consider writing integration tests in a language that is not embarrassingly slow. There's no reason you need to write your "make this request, get this response" tests in Ruby.

# ? May 4, 2018 23:33

necrotic: Aug 2, 2005; I owe my brother big time for this!

He said model tests so that's entirely unrelated. And using a fast language to test against your slow language is pointless.

# ? May 4, 2018 23:50

xtal: Jan 9, 2011; by Fluffdaddy

necrotic posted:

He said model tests so that's entirely unrelated. And using a fast language to test against your slow language is pointless.

You can't do anything about slow unit tests unless they're poorly coded. So if you have a slow test suite, the answer is clear. Using a fast language to test against a slow language makes perfect sense when the problem you're solving is slowness.

Maybe try running the unit tests in JRuby or something. It is an unsolved problem that running tens of thousands of tests in a slow language is slow.

# ? May 5, 2018 00:07

Gmaz: Apr 3, 2011; New DLC for Aoe2 is out: Dynasties of India

How would the language the tests are written in make a difference in speed if the executed code is still in a slow language?

EDIT: Also from my experience if it takes you 5 seconds to run a single model spec, then either your test is not written well or the model should be refactored. Slowness of the language is like the last thing I would think about.

Gmaz fucked around with this message at 00:49 on May 5, 2018

# ? May 5, 2018 00:46

Nybble: Jun 28, 2008; praise chuck, raise heck

Weird things might be happening with the Database or the Before section too. See if anything is being repeated that doesn't need to be.

# ? May 5, 2018 00:55

xtal: Jan 9, 2011; by Fluffdaddy

Gmaz posted:

How would the language the tests are written in make a difference in speed if the executed code is still in a slow language?

Which number is larger, 2 + 2 or 2 + 1?

# ? May 5, 2018 01:05

Gmaz: Apr 3, 2011; New DLC for Aoe2 is out: Dynasties of India

What if it's more like 2 + 0.001 vs 2 + .01, not counting the time it would take to rewrite all those tests. Again from my experience people often blame the slowness of the language first when it's usually the last bottleneck - at least when it comes to web applications.

# ? May 5, 2018 01:11

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 20:25

xtal: Jan 9, 2011; by Fluffdaddy

Other people mentioned that if your tests are taking 4-5 seconds to run they are probably coded poorly. This is correct, so you should tackle those low-hanging fruit first if they are applicable. Once the code is perfect, there is still the problem that each and every Rails app eventually has a slow test suite. That blames the language last, not first. Slow tests are an inevitability for Rails apps of medium-to-large scale.

Having said all of this, I will make an edit to suggest that the person in question benchmark the slowest tests and post those for inspections.

xtal fucked around with this message at 01:24 on May 5, 2018

# ? May 5, 2018 01:19

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ruby on Rails Love-In

«‹›178 »